12 ms·

Tests Are Overhyped

149 points by sturgill 14y ago

coolsunglasses 14y ago
I tended to do API-centric functional testing (does it spew JSON with the structure and data I expect?), and found it to be a time-saver in that it was faster to write a test and re-run it to check output than to manually use the web app or make the calls myself. However if the test exceeded this cost/benefit metric where it wasn't really helping me get the feature written, out the window it went. Helped when I went to refactor/fix fairly major chunks of the backend as all those tests from back when I did initial development were still there. It wasn't really "test first" because I didn't know what to test for until the basics of the API endpoint were in place. This was Python if it matters (default unittest2). I do mostly Clojure when I have the choice lately. It's not even really a matter of "does it need tested?", although you should be asking that question and building up the coverage for the critical bits. For me it was a question of, "is this going to save me time/sanity?" I advocated tests to the other engineers at my startup only when they were experiencing regressions/repeat bugs. I left them alone about the lack of test discipline otherwise. My Clojure code tends to "just work" (after I'm done experiencing insanity and disconnection from reality in my REPL) to the degree that I mostly write tests when I'm making a library or touching somebody else's Git repo. This is all fitting though. I use Clojure instead of Haskell precisely because I'm impatient, lazy, etc. Would kill for a "quickcheck" for Clojure though. This whole debate has a whiff of excluded middle (we have 3 UND ONLY 3 TESTS <---> TDD ALL THE TIME!), not to speak of people not mentioning how tests can simply...save time sometimes.
ericcholis 14y ago
I think finding a critical balance what to test and what to skip is often overlooked.
scootklein 14y ago
Care to comment on your experience with larger code bases? The content of your post seems short-sighted, and there's an exponential function of complexity increase as LOC and developer headcount both go up. You're right people pay for features, but lagging a little at the beginning to establish good TDD culture pays off in spades later on. Shipping product is something you have to do continuously, and you arguably create more value as time goes on, so ensuring you can continue to ship product in a timely manner is a great thing for organizations.
- ori_b 14y ago
  I'm not the original poster, but I find that integration tests have a far larger payoff than unit tests, in general. A good release process that tests differences in behavior between a test system and the current version of the service in production is also valuable. Being able to test that the whole system works as intended gives a better return on investment, in my experience, than testing small bits in isolation. The errors are, often as not, in the glue between the small bits.
  - t_hozumi 14y ago
    Agree, integration test is cost-effective. And I would like to rephrase "integration test" as "test from outside as much as possible".
  - plorkyeran 14y ago
    The main advantage I've found from testing little bits in isolation is that it tells you a lot more about where exactly the bug is. However, there's no reason why you need to write them in advance to get this benefit. The general workflow I've grown fond of is to write mostly higher-level tests that test the system as a whole, and then only write fiddly unit tests when there actually is a bug to fix. Those unit tests then stick around, but I don't feel bad about blowing them away without replacing them if the thing being tested changes significantly enough to make them useless.
  - chii 14y ago
    unfortunately, integration tests tend to take a longer time to run. unit tests tend to run instantaneously.
  - pkolaczk 14y ago
    Unit-tests work great for us for testing some well isolated pieces of complex logic with a simple API (e.g. algorithmic stuff). But we don't unit-test things simple like getters or setters or code that is glue between components and just delegates real work to some other components. For this, we do functional / integration testing and it plays well.
- 14y ago
ultimoo 14y ago
An ex-colleague of mine used to put it well in saying: "It's best to be pragmatic rather than dogmatic while testing and pair programming."
- machinagod 14y ago
  I think that can be easily generalized to: "It's best to be pragmatic rather than dogmatic."
  - artsrc 14y ago
    It is pragmatic to be dogmatic in a context where there is a a temptation to cut corners which will result in non-obvious, severe negative consequences.
    - contingencies 14y ago
      Ahh, but is it not dogmatic to assert a statement is always true for one class of situation? Accepting imperfection, are we not pragmatists? 道可道非常道 (The Tao that can be told is not the eternal Tao) 名可名非常名 (The name that can be named is not the eternal name.)
swalsh 14y ago
Get halfway through a project, realize you have to make a big change, then make it. Without tests i'll guarantee you'll watch your stability plummet. With tests, you might just go home at 5 pm. Alternatively don't make the change, and deal with a problem in your design. I've seen it more than a few times to have become convinced that there's a lot of value when striving to cover as much of your code as you can.
- to3m 14y ago
  If you have a statically-typed language - and it doesn't even have to be a good one, with C++ being fine, in my experience - then making such changes is usually a case of making the change, or perhaps removing the bit of code you intend to change, compiling, and seeing what happens. What happens is usually that you have 14,000,000 compile errors. Well, congratulations! That's the difficult part over. Now it's time to relax! Start at error 1, and fix it. Repeat. Every now and again, build. Once it builds again, it will usually work; if not, it's always something quite simple. If you have enough asserts in your code, then you can be doubly confident that there's nothing wrong with the new code. I've had good success with this approach, for all sorts of changes, large and small. I've given up being surprised when the result just builds and runs. I have no real idea, how you would do this reliably in a large program written in something like python. Sure, you'd fix up your tests easily enough... but then what? Don't you have a program to maintain as well? :) Jonathan Blow wrote about this years ago: http://lerp.org/notes_4/index.html http://lerp.org/notes_4/index.html
  - Evbn 14y ago
    If you have 140k compile errors, you might need to factor out repeated idioms into functions.
    - pkolaczk 14y ago
      140k is probably huge, but when talking about reafactoring, error numbers can go quite high. Refactoring is usually not just replacing a few private methods.
    - 14y ago
lbarrow 14y ago
I use a strongly typed functional language, so I don't need to write tests. If my code compiles, it's correct.
- _ak 14y ago
  Poe's law applies here.
  - rhizome31 14y ago
    Thanks I didn't know about Poe's law. My first thought was that it was a joke and a rather good one, but then seeing replies taking it seriously I started to doubt :)
  - lbarrow 14y ago
    ;)
- tikhonj 14y ago
  This is not accurate unless you use a nice dependently typed lanugage like Agda or Coq. It is true that you need far fewer tests, but hardly none at all. This is particularly evidenced by the fact that Haskell--certainly a "strongly typed functional language"--also has some of the best testing facilities of any language I've seen. QuickCheck simply can't be beat, and you can cover other parts of the code with more standard tools like HUnit. Now, there is some code--only very polymorphic code--where the type system is strong enough to give a proof of correctness. For that sort of code, which you're only likely to encounter if writing a very generic library, you can get away without testing. But that is not even the majority of all Haskell code! And even there you have to be careful of infinite loops (e.g. bottom). Comments like this make functional programmers sound much more arrogant and clueless than they really are.
  - pbiggar 14y ago
    If you try hard enough, you can make code that doesn't work in Haskell too!
alexjarvis 14y ago
I'm sorry, but I've seen both sides of the coin here and think that in the right situations, having a suite of automated-tests to give you confidence in an application is invaluable. Especially when you consider the alternative, which is hours wasted debugging. Sure writing tests can be hard, but it will make you write better, more modular code. If you really do need to just get something out the door that quick, then you should just be prepared to accept the consequences of no tests, or put some in later for maintenance sake.
dsdjung 14y ago
Test early to minimize the cash investment you are worried about in test.
raganwald 14y ago
My experience is that you can write an essay like this about any development practice. You can cite your experience ignoring it, you can say some nice things about the importance of experience and priorities and managing trade-offs, and you can satirize people who are dogmatic to the point of becoming cult-zombies. It appears as if such an essay providing great advice. But it it? What is the reproducible, objective process or behaviour this essay is advocating? If I give this essay to five teams and then look at what they are doing to follow its advice, will they all be doing the same thing? What results does this essay suggest we will obtain from following its advice? How can we be sure it will be better than the results we would obtain from mindlessly chaining ourselves to the red/green light? For better or for worse, the red/green light protocol is specific. As such, it can be criticized. A specific protocol has edge cases. You can argue slippery slopes. It's a stationary target. Whereas, vague and nonspecific generalities are more difficult to criticize. There are no details, so how does one identify edge cases? It's great to hear about this one specific anecdote. beyond that, what am I supposed to walk away with besides the knowledge that slavishly following a process can be bad?
- sturgill 14y ago
  I'm not sure that there is a sure-fire way to quantify what tests are or are not necessary. In my opinion, this is something that comes with experience and is more of an art than a science. But I'm okay with that. If all you got out of the article was that slavishly following a process (like TDD) can be bad, I'll consider that a win for my writing. Because my general feel of the community is that testing is some sacrosanct practice that must be adhered to without question. And I question that. I write tests. I'm not anti-test. I just imagine I write far fewer tests than most. In my opinion we should constantly question whether a certain practice (in this case, writing a specific test) is worth it. I think that's a healthier approach to development than blind red/green light.
  - trcollinson 14y ago
    I very much agree that knowing when to test and when testing is not needed is an art, not a science. I also hope that your readers know the difference between writing tests and writing code using tdd. My biggest problem with your statements comes down to the idea that you claim tdd can be bad (particularly, it would seem, for startups) and yet you do not give an alternative which has equal or greater benefits without the perceived overhead cost. You say we should constantly questions whether certain practices are worth it in stead of blindly following red/green lights. Fair. But how will questioning practices give you the knowledge that your code is not broken when a junior developer makes a change "somewhere"? How will questioning practices allow you to better refactor when you pivot quickly? I have worked in many startups and I know time is money and bank roll is king. But I believe that tdd allows you to keep those financial concerns in the black longer. If not tdd, what is an alternative method? "not writing code in tdd", is certainly an alternative one, but is it a good one?
biot 14y ago
My approach to tests is that where you control all dependencies, do tests for core functionality and any tricky parts. After all, you can always hot-fix any breakage since everything is under your control. However, where you have external dependencies (eg: an API that third parties use) you should have fairly robust tests to ensure you are not introducing any regressions.
iow 14y ago
It's true that some tests might get redundant when your functionality changes, but you don't know which part you will throw away at the point you're writing it. Better have that safety net for the part that will need to continue to work after you refactor.
mdkess 14y ago
In my admittedly limited experience, unit tests are way overhyped, especially when things like mocking are brought into the mix. It's easy to end up with a test that is not about correctness, but "did line 1 get called, did line 2 get called, etc.". Then you change the implementation, and 20% of your test cases break. That's not to say that they are valueless, but that I think unit tests should be used pretty sparingly. Where I've found a ton of value has been in writing almost artificial-intelligence driven integration tests. Write a bot that uses your service in way, as fast as possible. Run fifty of these bots simultaneously, and see what happens. Then have some way to validate state at various points (either by tallying the bots actions, or sanity checks). Bugs will come fallout out of the sky. Then, in the future, when you get a bug, the challenge becomes to update the integration test bots behavior so that they (preferably quickly) can reproduce the bug. I mean, I think that this is dependent on the domain of your software, but I think it's a good strategy for many areas.
- gknoy 14y ago
  If we're only using unit tests to see if line N gets called, I think we're doing it wrong. Instead, we want to use unit tests to tell us if the answer we get is correct -- while exercising line N. e.g.: def test_foo(self): result = get_foo(bar=12) self.assertEquals(result.category, 'FOO') self.assertEquals(result.name, 'my-foo-12') self.assertEquals(result.unit_price, 97.12) This lets you verify that this particular branch (when bar=12) is executed, and that your results are as you expect. If you change some of the underlying calculations, things can break (as you get different answers), but then you at least have a test that lets you ensure that changing the answers is what you want to do. Sometimes, you want to change the way you calculate something and get the same answer, after all.
  - mdkess 14y ago
    So the question is, what if bar = 12 means "retrieve foo with ID 12 from the FooService", and we want to make sure that get_foo does something sensible when the server returns an error code. So what do you do? Typically one writes a mocked up service that returns the expected results, pass it in somehow, and write your tests like you did above. So if you have a lot of services, you end up doing what I said - testing lines of code. Here you're testing that FooService gets called. If it also called BarService, you'd be testing that BarService gets called. And so on. But then later, we decide that FooService is no good - we want, nay demand - a FooService2, with a new API. However, get_foo is to behave the same. So what now? Only one choice - update all of the tests to have a mocked up FooService2. As the code base grows, I find this becomes annoying to maintain (although to some extent a necessary evil, of course). The alternative I'm saying is that instead of spending a ton of energy unit testing get_foo, hammer on it and write good state validators. Your unit test shows that get_foo for bar = 12 returns a unit_price of 97.12. Great, but I don't think that's as interesting as building a fake database (so we know what the expected returns are), and then making 50,000 requests per second with random data for 12 hours, and then through runtime assertions and as a post-process validating that the service actually did what it was supposed to.
shubb 14y ago
I'm not necessarily disagreeing with the author, but I think if you are going to 'do agile', especially in a breaky language like C++, you need to do tests. Lots. The current shop I'm at is maintaining a huge code base, parts of which go back 20 years. Because test coverage is so low, there is a real reluctance to refactor. The first thing I did when I started here was to clean up the code, renaming miss-named variables to get it in line with the coding standard, adding autopointers here and there to head off memory leaks. By gosh, I nearly got fired. If you are going to fearlessly edit your codebase, you need to know that regressions are going to be caught. You need automated testing.
azov 14y ago
tl;dr: Tests are overhyped because we once wrote tests that became irrelevant when the thing we were testing changed Really? I mean, tests might be overhyped, but so far it looks like the author draws incredibly general conclusions from some isolated incident. Why did the tests become irrelevant? Is their scoring algorithm now doing something completely different? Do they now have different use cases? Were they hard-coding scores when the actual thing that matters was, say, the ordering of the things they score? > Only test what needs to be tested well, thanks for the helpful advice :) Care to share what, in your opinion, needs to be tested?
- jaredsohn 14y ago
  >well, thanks for the helpful advice :) Care to share what, in your opinion, needs to be tested? I am not the OP, but I think unit tests are more useful for code that has a lot of edge cases (such as string parsing) and for code which causes the most bugs (as seen in black box testing or integration tests.) Also, some code is just easier to develop if you create a harness that runs it directly instead of having to work your way to the point in the program where it would execute it. If you do this, you might as well turn it into a test.
- jamesaguilar 14y ago
  I've heard this saying: "Test until fear turns to boredom." Then you need to make sure you get bored at a properly calibrated rate (i.e. not too soon given your business' criticality, not too late given your need to move quickly). For example, if a lot of expensive bugs during final cause analysis are found to be preventable via unit tests, then your calibration is probably tuned to being bored too quickly.
  - mmcnickle 14y ago
    That's an excellent saying. It's exactly the line I try to tread with testing. I don't know what to do with the angst of getting the tradeoff wrong though.
  - gfodor 14y ago
    This is a great line. I think it goes deeper though, in that after doing this for a while, if it hurts, you are probably doing it wrong. Writing code with too little tests hurts, because you are scared shitless. Writing code with too many tests hurts, because you are performing a religious rite not actually building something useful. The trouble is that you have to do this for a while to get to the point where you can both feel and recognize pain, or the lack thereof.
Skoofoo 14y ago
Behavior/test-driven development needs all the hype it can get. There are so many developers, even entire software development subcommunities (I'm looking at you, video game developers) who haven't written a single test in their lives and don't understand its value. Yes, it is time-consuming and invisible to your customers, but I imagine so is setting up a frame for a house instead of just stacking bricks. The structure, flexibility and peace of mind you get from a comprehensive test suite pays off when you have a 50-brick tall structure to put a roof on.
- rhizome31 14y ago
  In my experience test-driven development is not time consuming, even for small developments. I don't think we even need to say things like "yes it takes time now but it will pay off later". I have found that it saves time right from the beginning. It seems other people have a similar experience : http://blog.8thlight.com/uncle-bob/2013/03/11/TheFrenziedPanicOfRushing.html http://blog.8thlight.com/uncle-bob/2013/03/11/TheFrenziedPan...
RivieraKid 14y ago
I agree and this is not the only thing that is overhyped right now...
- antidaily 14y ago
  Also overhyped... trains, The Lumineers, IPAs.
bazzargh 14y ago
Funnily enough what the article describes is almost the perfect case for testing: "We wrote the first scoring algorithm...based on the red-green light." This sounds like it describes some heuristic weighting, which couldn't have been solved by types, but reasonable tests would have shown if the new algorithm weighted some special cases higher or lower than it should. The problem with an heuristic weight though, is that it's an heuristic, judged against other heuristics by taste not proof. The obvious testing approach, ensuring that the score for each test case retains the same order as you tweak the algorithm - is overtesting. You don't care about this total order; you more likely care about ordering of classes of things, rather than ordering within those classes; or simply that 'likely' cases follow an order. Hence, you hit far too many test failures. I'd agree that it's possible to overtest in general, but it's so easy to overtest heuristics that it needs called out as a special case, and it sounds like the problem here.
kosma 14y ago
If the author stopped at the title, he would be one hundred percent correct. It's easy to rant about unit tests. Don't fall into that trap. Be aware that it's a hype, and correct for that fact - and once you do, use your best judgement. In the end, it's you, the engineer, who's shipping the product - not the countless ever-arguing denizens of the internet.
trcollinson 14y ago
I find it interesting that the premise for this entire diatribe is that once some code was written, a pivot was made, and a number of tests then had to be thrown away (or reworked, or re-factored?). Recently I had a code base where we decided that due to a new delightful feature our customers were going to be quite pleased with we would need to switch out our old queuing system for a new one. In doing so well more than half of our huge test suite turned red. This told us two important things 1) that the queuing system touched a lot of areas of code we needed to think about and 2) where in the code the queuing system had touch points. Ultimately we were able to put in the new queuing system, fix the areas that were broken by the change, and have the confidence at the end of the process that we had not broken any of the areas of the code that were previously under test. (This does not mean that our code was bug free of course, only that the areas under test were still working in the prescribed way, but that is a discussion for a different article.) I believe that this would have taken a team of people weeks to do previously. I was confident that the change was ready after only 3 days with 2 developers. I would not trade my tests. There is a cost associated with everything, but I believe tests are the least costly way to get highly confident software built.
embwbam 14y ago
I'll take a crack at my own over-generalization, albeit one I think is more useful. Adding strict types (Typescript!) to your code base gives you way more maintainability than testing. Do it first, then if you still needs tests, write them.
lutorm 14y ago
The importance of testing obviously depends on how severe the consequences of shipping broken code is. If not testing means that some broken UI element on some webpage slips by, that's obviously easier to live with than if not testing means you blow up some $150M piece of hardware.
gfalcao 14y ago
This post is over-hyped: "the cool guy who ships products and applies to cool startups, but they only hire super talented and passionate engineers." --- The text above is just like the blog post: is unfounded and purely based on personal opinions. Having a strong test culture in a company of any size is important for a number of reasons, and all of them are founded by both books and articles, but also by AMAZING products. An engineer who has talent in writing readable and well structured tests is also someone who has a special skill in: creating robust systems, reverse-engineering existing systems and therefore good at being hired by a startup that has NO or very little test culture, which end up being hired by a company that has code that is buggy, error prone, coupled and hard to maintain, so that he can apply all this knowledge to clear crap written by people who doesn't like tests
kasey_junk 14y ago
Tests are not overhyped. They are under-understood. Unit tests are not regression tests are not integration tests. I've encountered tons of teams that don't seem to understand this. Unit tests are more than any other factor a design tool. Like any other design tool (uml, specification, etc), when the design needs to change, you throw them out. If it takes longer to design a system with unit tests than without them 1 of 2 things is true 1) you should not write unit tests 2) you should learn how to write unit tests.
PuercoPop 14y ago
>>> We wrote the first scoring algorithm at Goalee based on the red-green light. Within a couple of weeks we made our first algorithmic change, I think he actually mens an specification change. Tests would help you refactor in case you want to change the algorithm against the same specification. But yeah, is a flawed metric so it shouldn't be put on a pedestal.
spitfire 14y ago
So because his particular tool (or use of a tool) adds inconvenience or overhead means the entire concept is bunk? Don't confuse the thing with the concept. Tests are double checking your work. ("Yup. That condition I said would be true still is!"). The tighter and more lightweight we can get that feedback loop going the better out output and the more confident in our work we'll be.
kscaldef 14y ago
"We wrote the first scoring algorithm at Goalee based on the red-green light. Within a couple of weeks we made our first algorithmic change, and made several quick fix releases to update the scoring methods in the following weeks. By the end of the month, a whole section of our testing suite was almost completely worthless. In a world where cash is king, I wouldn’t mind having those dollars back." What I don't understand about comments like this is that a whole section of your code, both runtime / deliverable code and test code had become worthless. But, you only seem to view the discarded test code as wasted effort. Either the tests have value or they don't. And, if you write tests, and then discard the code they test, you'll likely also discard the tests. But, that doesn't change whether or not the tests had value, nor whether the new tests that you'll write for the new code have value.
- dspeyer 14y ago
  > What I don't understand about comments like this is that a whole section of your code, both runtime / deliverable code and test code had become worthless Not so. The code demonstrated that the first algorithm wasn't good enough and provided the experience needed to write the second one. The tests (hopefully) made the first algorithm's code maintainable, but it turns out there was no need to maintain it.
mstrem 14y ago
Depending on the context, I fully agree (most stuff here is about web development so in general shipping a broken web page is fine - it wont cause too much damage even when it happens, and it can be fixed quickly). In Safety Critical Software however, I hope tests have been written for every single change, and everything has been checked over and over again. Tests become a lot more valuable and simply having to re write them, may make you think of problems you have introduced in the code.
sippndipp 14y ago
Sounds not very sustainable to me. Ruthless and speedy refactoring is only possible with tests. Better onboarding of new team members is only possible with tests. Updating external libraries is only possible with tests. How many hours did I spent in projects w/o tests figuring out what's going on... things that an easy test could have told me in the first place. In a world where cash (=time) is king, I wouldn’t mind having those dollars back.
doki_pen 14y ago
Like anything in life, you have to make a judgement. There is no silver bullet. Tests are very painful to maintain in all but the most well understood problems. There is a ton of great software we all rely on that doesn't have much testing. Having a ton of people rely and test beta versions of your code is much more valuable then any test suite. Dare to think.
FreeFull 14y ago
It's a good idea to record invariants about your code and your problem. Tests are one way, and there are some other ways. It's a good idea to have the checking of invariants be automatic somehow. Any assumptions about the output of your code that you make but don't record in a checkable form, are something that could potentially be silently broken by a change in the code. And then you're wasting your time debugging rather than solving the problem you're working on.
jamesbritt 14y ago
[I]f any portion of your site changes (pivot, any one?) you’re now in a position of having to rewrite large portions of your tests. And the larger the pivot, the more useless your old testing becomes. I'm under the impression that the tests are written first. If you are writing new code, then first there's a (failing) test, then there is the new code. If you are altering existing code, then pivot or not if something breaks you need to see if the reason is broken code or an inappropriate test, and you fix it before moving on. In any event this should be a gradual process. How common is it that large slabs of code gets changed while ignoring failing tests, or large slabs of new code gets added while skipping the tests? Is it that TDD is, in actual practice, unsustainable, or is the problem with not adhering to TDD? Or something else altogether?
pbiggar 14y ago
Amen! I run https://circleci.com https://circleci.com, where we make all of our money by actually running tests for people, and I still believe this. The goal of your software is to achieve goals, typically business goals. Often the software itself is already one or two steps removed from the value provided to customers. Tests are an extra step removed at least. I wrote a similar piece here: http://blog.circleci.com/testing-is-bullshit/ http://blog.circleci.com/testing-is-bullshit/
- Evbn 14y ago
  Of you don't care whether your CI tool works, I won't care either, because I won't use it.
  - pbiggar 14y ago
    Perhaps I missed where I said I didn't care - where did you see that?
altcognito 14y ago
I find unit tests written after the fact to be pointless. TDD makes a lot of sense because it forces you to think about how you are going to write your code. I like that. If you're writing tests after the fact, integration tests have a much more satisfying finish, and I can't imagine going without them (unless they are painful to implement)
rdtsc 14y ago
Tests for software development == Crimes for political agenda. This is one of those issues you are supposed to lie about. Tests are like crime in politics -- no politician is going to say "I am soft on crime, I think we should reduce sentences." So crazier and crazier laws are created. Mandatory minimum sentences for possessing small amounts of drugs, all kind of craziness. People know it is crazy, but nobody is capable of speaking up and remain standing. Same thing with tests. Nobody can say "fuck it, stop writing tests, the customer will never run this code, or this is too fucking obvious, test stuff that matters". That is considered irresponsible. Everyone is supposed to worry about not having enough tests. Now there a subtle difference and that is when it comes to shipping on a schedule and in many companies tests are ignored -- BUT silently. Everyone loves talking about tests but try and tell you boss you spent 2 weeks on adding tests to your code. They might not like. Try to double the time you promise to do a feature by saying I need to write tests for it. Or if you are given free rein try saying I won't do all the new awesome features, I'll write more tests -- it will be approved probably but everyone in the end will praise the guy who chose to work on and deliver features -- even though they might be completely buggy and unusable. So that is the other side if you will.
- StavrosK 14y ago
  > Same thing with tests. Nobody can say "fuck it, stop writing tests, the customer will never run this code, or this is too fucking obvious, test stuff that matters". That is considered irresponsible. Everyone is supposed to worry about not having enough tests. I don't know about everyone else, but I definitely worry about having too many tests. For every test I write, I have to weigh how useful the test is versus how likely it is that the thing it is testing will change. If its usefulness is overshadowed by the likelihood that it will be a burden later on, it does not get written.
  - rdtsc 14y ago
    Yap. It is very easy to write bad tests -- tests that test too much, tests that test the wrong thing, tests that are too fragile (very timing dependent, they break intermittently and suck up resources in tracking down the cause), tests that are not repeatable (someone inserted random input generation and seeded the generator with the time function).
huntedsnark 14y ago
Another day, another link-bait, faux controversy post about testing on HN. TLDR: Writing bad tests makes for a bad time. Do we need a whole article to cover that? As a side note, all this recent hand wringing over testing reminds me so much of the time when the new breed of version control software was coming out, it was the same kind of accusations about things being "overhyped" and "you can just email a tar file!"
donebizkit 14y ago
I have a long experience in enterprise software and I agree with the premise. There are two kinds of test units: workflow and functional. 1 - Workflow test units are a waste of time because no single test unit stays valid when there is a change. In other words, whenever we added/removed steps in the workflow, 99% of the time we have to change the test unit to fit that new workflow which breaks the concept of "write once, test all the time" concept. In my experience, having proactive developers who test areas around the workflow that they changed is much faster and reliable. 2 - Functional test units are great. They test one function that needs certain parameters and is expected to spit a certain output i.e function to calculate dollar amounts or do any king of mathematical operations. However, these functions tends to stay unchanged during the lifetime of a project. Therefore, the test units are rarely run. From my experience workflow changes/bugs represent 80% of the problems we face in enterprise software. Functional changes/bug are rare and can be detected quickly. This is why I agree with the author premise that unit testing is overhyped.
stcredzero 14y ago
[X] is overhyped != [X] is useless
trustfundbaby 14y ago
In total agreement with the OP. I think it just goes to the way human beings handle original ideas, first they fight them, then they embrace them, then they take them to ridiculous extremes as they try to substitute rules for common sense in applying them. You can see it in politics, religion and almost any really popular area of human endeavor. Testing falls in the same category, I have had interviewers look me in the eye and in all seriousness, declare that developers who don't write tests for their code should be fired, or that their test suites cannot drop below x% of code coverage. Dogma is a horrible thing to afflict software teams, whether it is pair programming, or mandatory code reviews, if there are no exceptions to the rule or situations where you don't have to apply it, its probably a bullshit rule IMO. Me, I like to ship shit, and I like to look at tests as a way to help me ship shit faster, because the less time I spend fixing regressions the more time I can spend actually getting more code/features out that door. So my only metric for writing tests is this ... "is this going to help (me or a team member) not break something months from now, when I change code somewhere else". I honestly don't care about anything else.
- Evbn 14y ago
  Tests certainly interfere with your ability to ship shit.
teek 14y ago
My general rule: If you are planning on some sort of reuse (API, library, protocol, etc) a test spec is a very good idea if only to hash out practical use of the interface and identify caveats in the interface spec. Once released, the test spec is great for answering the question "did I accidentally break something that used to work?" thus vastly speeding up the bug-fix process for underlying components or widely used components. If you are writing an end-user app, or a service that will be used non-programmatically (such as a GUI or web app), then a test spec is not required and probably a waste of time as things will likely change too quickly.
GhotiFish 14y ago
I don't know what to do with tests, to be honest. I am a very inexperienced developer, and the general sentiment is "knowing what to test is a learned skill" damn. I could use some opinions actually. The current workplace arguments about what we should or shouldn't test go like this: "We write tests to establish specifications, then we write code to meet those specifications. We should have tests covering every aspect of the system as a way to assert that things are meeting the designed specs." "but our model objects have allot of tedious detail. Why should we write tests about what the name field can and can't accept? It's a max length of 32, that's it. Asserting that the constraint fails is beyond tedious and a waste of our time. Most of the code we'd be testing is auto-generated. Are we running tests on our framework or our code?" "Some of this does seem like a waste of time, but what about when someone changes something as part of the development cycle? eg. Someone needs to remove validation from the name field to test other parts of the systems behavior with an overly large name. If we had a complete testing system in place it would alert us that the model isn't meeting the spec. We've already had instance where temporary code to bypass authorization was accidentally pushed to the develop branch." On one end of the spectrum there is this base level of tests that are actually REALLY useful when developing features. Doing simple testing has helped me catch off by one errors and other kinds of bugs, I can stand by my code confidently. These kinds of ad-hoc in development tests don't cover everything, but they're useful. They're like a substitute for step through debugging to verify code. On the other, I don't know how big of an issue bug regressions will be, we've heard other developers suffering for lack of a solid testing base to detect problems. Without a full battery of tests asserting everything we know about the spec, there's no way we will catch many regressions. HN what do you think?
- columbo 14y ago
  > I don't know what to do with tests, to be honest. ... HN what do you think? Well, since you asked! The most valuable tests in any system is functional tests. Unfortunately most tests are unit tests that give little value. Here's something that I encountered recently in the wild. Imagine a situation like this, a calculation process with three distint black boxes of code. Block A - Pull data from 3rd party source Block B - Distribute data to individual nodes (multi-threaded environment) Block C - Individual nodes processing data Each one of these blocks is around fifty thousand lines of code, each one of these blocks has hundreds if not thousands of unit tests. Every line of code is tested by a unit test. Yet there is a very real and dangerous bug in the system. Why? Because the object passed from B-C has a transient variable and when it is serialized and deserialized that value is lost and reverted to the default value. Now most places leave this value as the default, so it's only one out of a thousand customers that have this problem, but when they do it's a huge problem. Functional tests is the process of saying "I put meat/spices on this side and sausage comes out the other" it doesn't try to determine the RPM of the grinder, the number of bolts, the rotation of the gears, or the type of metal used to build the sausage-maker. It is simply the process of determining that meat becomes sausage. Functional tests have high value and are absolutely worth knowing. In most CRUD applications Unit Tests generally wind up testing that the universe still exists as expected (true is still equal to true, zero is not one, and a null value is not the same a string, oh happy days the system works!). That's not always the case. If you are working on something that is more advanced, say something that should measure the chance a hurricane might knock down a given 3d model you'll wind up having a huge stack of unit tests, and these will be very valuable. More often than not you'll know they are valuable as they'll be spent proving the science behind the application, and not simply that the application 'does something'.
ryan-allen 14y ago
I think the hype of tests is driven by agencies that charge by the hour. One thing I have noticed is that on an existing system especially, you can put a very inexperienced developer down and instruct them to 'write tests for untested functionality', and do so but produce neligible value. The same thing with green fields development. I've seen steaming piles of shit with huge test suites. Absolutely zero insight into the problem. No craftmanship at all, nothing interesting about the application. But a set of tests. It's like the suite is proof enough that there was a job well done. I fear that a lot of development is devolving into nothing more than superstition and hype, backed up by agencies that like to bill a lot and amateurs who need a justification for their timelines and ineptitude.
pesenti 14y ago
Seeing how many people agree with the post I am left wondering who the commenters on HN are. With a large code base and actual customers, comprehensive unit testing is by far the most cost effective way to create and maintain software. This is not being dogmatique, it's the shared experience of the majority of established software engineering firms. In the case of my company, we experienced so much growing pains that our productivity almost came down to zero before we switched to TDD. Many developers in our group were skeptics (including me) but today you won't find one of us arguing for less testing. The kind of large pivot that the author refers to is only possible when you don't have established customers and you have a minimal product. You may as well call it prototyping. And prototyping with or without tests is indeed more a matter of taste than effectiveness.
laureny 14y ago
His beef is more with unit tests than functional tests, and I can't disagree with that. Functional tests code that will directly impact your users while unit tests are merely there to make the developer's life more comfortable.
dvt 14y ago
Even though I can sympathize with the tone of the OP, I think his argument cannot be generalized. Consider the following scenarios: A one-person engineering team. This is some 20-something working on a start-up in his parents' garage or the 40-something code hobbyist out there that likes writing fun little apps every now and then. I would argue that there is virtually no need for testing unless you're either bad at writing code or inexperienced. I've written dozens of applications (in various languages) and once you get into the swing of things, you don't need TDD to actually "test." Debugging/edge cases/etc. just becomes a natural process of writing code. A two-person engineering team. This is a secondary stage in every start-up -- sometimes, start-ups even are founded by two engineers. Here is where TDD starts being important. I may be an expert coder, but if my partner isn't that great (or if our code doesn't work well together), not testing can be a huge nightmare. But the impact is still relatively small. Bugs are easy to spot as both engineers are (or at least should be) incredibly familiar with the entire codebase. Three-person to 10-person engineering team. Here is where things get really tricky and TDD becomes integral to meeting deadlines and actually, you know, putting out products. You've got Jim working on some financial stuff while Bob is optimizing some JPA code. At the same time, Susan is implementing a couple of new features. Having a test suite ensures that behavior will remain predictable given non-centralized iterations. Without TDD, large teams would be fixing bugs more than they would be writing code (most teams probably do that anyway -- not that it's a good thing). 10+ people; open source projects, etc. When you have more than 10 people working on a project, I think that TDD is simply a necessity. There are simply too many moving parts. Without a cohesive test suite, having this many people mess around with code will undoubtedly break stuff. Ironically, I think that small teams put too much emphasis on TDD, whereas teams in larger companies (AT&T and IBM, for example) put too little.
camus 14y ago
tests should be fast and easy to write , that's why any library should be bundled with things that make it easy ( null classes , mocked classes , ect...).
digitalWestie 14y ago
Can we please stop referring to 'testing' as if it's the same thing as TDD?
pkolaczk 14y ago
No amount of tests can help you if don't know how to solve the problem. And the better you know how to solve the problem the less tests you really need. ;) Old, but quite relevant: http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-solvers.html http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-s... Some sane amount of testing is good. However, I'm not very convinced writing tests first is a good idea. I saw a few programmers practicing this and when writing code they often concentrated too much on just passing the damn test-cases instead of really solving the problem in a generic way. Their code looked like a set of if-then cases patched one over another. Therefore, if they missed an important case in their testing suite, they had 99% chances for a bug. Once I ended up writing a separate, more generic implementation to validate the test suite. And it turned out there were a few missed/wrong cases in the test suite.
dwhelan 14y ago
There seems to be a sense that writing tests slows you down. And that is understandable. However, I find that tests helps me move in small steps and provide a feedback loop that helps me stay in a groove moving towards the finish line. And they help me move faster. I think we as developers need to continually learn how to write better tests that provide deeper value and allow us to move more quickly. I find that when testing is tough it is often that the underlying design is deficient. And tests shine a light on code and design smells. Discarding tests could be valuable if the tests have outlived their value. It might also be the case that excessive test maintenance is telling you something about your production code.
TeeWEE 14y ago
Man I like this article so much. I worked at a company that aimed at 90% test coverage. Without taking into account how important the piece of code under test was. I think the 80/20 rule applies here. Only 20% of your code does 80% of the work. Focus on writing tests for those 20% then your time is much better spent.
Paddy3118 14y ago
Without due scientific process to back up TDD we are left with such arguments having equal validity as TDD. Without a scientific proof of TDD being superior then we are left with mere opinions on both sides. NAARP! http://paddy3118.blogspot.co.uk/2013/02/no-attempt-at-rigorous-proof.html http://paddy3118.blogspot.co.uk/2013/02/no-attempt-at-rigoro...

149 points by sturgill 14y ago

I tended to do API-centric functional testing (does it spew JSON with the structure and data I expect?), and found it to be a time-saver in that it was faster to write a test and re-run it to check output than to manually use the web app or make the calls myself. However if the test exceeded this cost/benefit metric where it wasn't really helping me get the feature written, out the window it went. Helped when I went to refactor/fix fairly major chunks of the backend as all those tests from back when I did initial development were still there. It wasn't really "test first" because I didn't know what to test for until the basics of the API endpoint were in place. This was Python if it matters (default unittest2). I do mostly Clojure when I have the choice lately. It's not even really a matter of "does it need tested?", although you should be asking that question and building up the coverage for the critical bits. For me it was a question of, "is this going to save me time/sanity?" I advocated tests to the other engineers at my startup only when they were experiencing regressions/repeat bugs. I left them alone about the lack of test discipline otherwise. My Clojure code tends to "just work" (after I'm done experiencing insanity and disconnection from reality in my REPL) to the degree that I mostly write tests when I'm making a library or touching somebody else's Git repo. This is all fitting though. I use Clojure instead of Haskell precisely because I'm impatient, lazy, etc. Would kill for a "quickcheck" for Clojure though. This whole debate has a whiff of excluded middle (we have 3 UND ONLY 3 TESTS <---> TDD ALL THE TIME!), not to speak of people not mentioning how tests can simply...save time sometimes.

ericcholis 14y ago

I think finding a critical balance what to test and what to skip is often overlooked.

scootklein 14y ago

Care to comment on your experience with larger code bases? The content of your post seems short-sighted, and there's an exponential function of complexity increase as LOC and developer headcount both go up. You're right people pay for features, but lagging a little at the beginning to establish good TDD culture pays off in spades later on. Shipping product is something you have to do continuously, and you arguably create more value as time goes on, so ensuring you can continue to ship product in a timely manner is a great thing for organizations.

ori_b 14y ago
I'm not the original poster, but I find that integration tests have a far larger payoff than unit tests, in general. A good release process that tests differences in behavior between a test system and the current version of the service in production is also valuable. Being able to test that the whole system works as intended gives a better return on investment, in my experience, than testing small bits in isolation. The errors are, often as not, in the glue between the small bits.
- t_hozumi 14y ago
  Agree, integration test is cost-effective. And I would like to rephrase "integration test" as "test from outside as much as possible".
- plorkyeran 14y ago
  The main advantage I've found from testing little bits in isolation is that it tells you a lot more about where exactly the bug is. However, there's no reason why you need to write them in advance to get this benefit. The general workflow I've grown fond of is to write mostly higher-level tests that test the system as a whole, and then only write fiddly unit tests when there actually is a bug to fix. Those unit tests then stick around, but I don't feel bad about blowing them away without replacing them if the thing being tested changes significantly enough to make them useless.
- chii 14y ago
  unfortunately, integration tests tend to take a longer time to run. unit tests tend to run instantaneously.
- pkolaczk 14y ago
  Unit-tests work great for us for testing some well isolated pieces of complex logic with a simple API (e.g. algorithmic stuff). But we don't unit-test things simple like getters or setters or code that is glue between components and just delegates real work to some other components. For this, we do functional / integration testing and it plays well.
14y ago

ultimoo 14y ago

An ex-colleague of mine used to put it well in saying: "It's best to be pragmatic rather than dogmatic while testing and pair programming."

machinagod 14y ago
I think that can be easily generalized to: "It's best to be pragmatic rather than dogmatic."
- artsrc 14y ago
  It is pragmatic to be dogmatic in a context where there is a a temptation to cut corners which will result in non-obvious, severe negative consequences.
  - contingencies 14y ago
    Ahh, but is it not dogmatic to assert a statement is always true for one class of situation? Accepting imperfection, are we not pragmatists? 道可道非常道 (The Tao that can be told is not the eternal Tao) 名可名非常名 (The name that can be named is not the eternal name.)

swalsh 14y ago

Get halfway through a project, realize you have to make a big change, then make it. Without tests i'll guarantee you'll watch your stability plummet. With tests, you might just go home at 5 pm. Alternatively don't make the change, and deal with a problem in your design. I've seen it more than a few times to have become convinced that there's a lot of value when striving to cover as much of your code as you can.

to3m 14y ago
If you have a statically-typed language - and it doesn't even have to be a good one, with C++ being fine, in my experience - then making such changes is usually a case of making the change, or perhaps removing the bit of code you intend to change, compiling, and seeing what happens. What happens is usually that you have 14,000,000 compile errors. Well, congratulations! That's the difficult part over. Now it's time to relax! Start at error 1, and fix it. Repeat. Every now and again, build. Once it builds again, it will usually work; if not, it's always something quite simple. If you have enough asserts in your code, then you can be doubly confident that there's nothing wrong with the new code. I've had good success with this approach, for all sorts of changes, large and small. I've given up being surprised when the result just builds and runs. I have no real idea, how you would do this reliably in a large program written in something like python. Sure, you'd fix up your tests easily enough... but then what? Don't you have a program to maintain as well? :) Jonathan Blow wrote about this years ago: http://lerp.org/notes_4/index.html http://lerp.org/notes_4/index.html
- Evbn 14y ago
  If you have 140k compile errors, you might need to factor out repeated idioms into functions.
  - pkolaczk 14y ago
    140k is probably huge, but when talking about reafactoring, error numbers can go quite high. Refactoring is usually not just replacing a few private methods.
  - 14y ago

lbarrow 14y ago

I use a strongly typed functional language, so I don't need to write tests. If my code compiles, it's correct.

_ak 14y ago
Poe's law applies here.
- rhizome31 14y ago
  Thanks I didn't know about Poe's law. My first thought was that it was a joke and a rather good one, but then seeing replies taking it seriously I started to doubt :)
- lbarrow 14y ago
  ;)
tikhonj 14y ago
This is not accurate unless you use a nice dependently typed lanugage like Agda or Coq. It is true that you need far fewer tests, but hardly none at all. This is particularly evidenced by the fact that Haskell--certainly a "strongly typed functional language"--also has some of the best testing facilities of any language I've seen. QuickCheck simply can't be beat, and you can cover other parts of the code with more standard tools like HUnit. Now, there is some code--only very polymorphic code--where the type system is strong enough to give a proof of correctness. For that sort of code, which you're only likely to encounter if writing a very generic library, you can get away without testing. But that is not even the majority of all Haskell code! And even there you have to be careful of infinite loops (e.g. bottom). Comments like this make functional programmers sound much more arrogant and clueless than they really are.
- pbiggar 14y ago
  If you try hard enough, you can make code that doesn't work in Haskell too!

alexjarvis 14y ago

I'm sorry, but I've seen both sides of the coin here and think that in the right situations, having a suite of automated-tests to give you confidence in an application is invaluable. Especially when you consider the alternative, which is hours wasted debugging. Sure writing tests can be hard, but it will make you write better, more modular code. If you really do need to just get something out the door that quick, then you should just be prepared to accept the consequences of no tests, or put some in later for maintenance sake.

dsdjung 14y ago

Test early to minimize the cash investment you are worried about in test.

raganwald 14y ago

My experience is that you can write an essay like this about any development practice. You can cite your experience ignoring it, you can say some nice things about the importance of experience and priorities and managing trade-offs, and you can satirize people who are dogmatic to the point of becoming cult-zombies. It appears as if such an essay providing great advice. But it it? What is the reproducible, objective process or behaviour this essay is advocating? If I give this essay to five teams and then look at what they are doing to follow its advice, will they all be doing the same thing? What results does this essay suggest we will obtain from following its advice? How can we be sure it will be better than the results we would obtain from mindlessly chaining ourselves to the red/green light? For better or for worse, the red/green light protocol is specific. As such, it can be criticized. A specific protocol has edge cases. You can argue slippery slopes. It's a stationary target. Whereas, vague and nonspecific generalities are more difficult to criticize. There are no details, so how does one identify edge cases? It's great to hear about this one specific anecdote. beyond that, what am I supposed to walk away with besides the knowledge that slavishly following a process can be bad?

sturgill 14y ago
I'm not sure that there is a sure-fire way to quantify what tests are or are not necessary. In my opinion, this is something that comes with experience and is more of an art than a science. But I'm okay with that. If all you got out of the article was that slavishly following a process (like TDD) can be bad, I'll consider that a win for my writing. Because my general feel of the community is that testing is some sacrosanct practice that must be adhered to without question. And I question that. I write tests. I'm not anti-test. I just imagine I write far fewer tests than most. In my opinion we should constantly question whether a certain practice (in this case, writing a specific test) is worth it. I think that's a healthier approach to development than blind red/green light.
- trcollinson 14y ago
  I very much agree that knowing when to test and when testing is not needed is an art, not a science. I also hope that your readers know the difference between writing tests and writing code using tdd. My biggest problem with your statements comes down to the idea that you claim tdd can be bad (particularly, it would seem, for startups) and yet you do not give an alternative which has equal or greater benefits without the perceived overhead cost. You say we should constantly questions whether certain practices are worth it in stead of blindly following red/green lights. Fair. But how will questioning practices give you the knowledge that your code is not broken when a junior developer makes a change "somewhere"? How will questioning practices allow you to better refactor when you pivot quickly? I have worked in many startups and I know time is money and bank roll is king. But I believe that tdd allows you to keep those financial concerns in the black longer. If not tdd, what is an alternative method? "not writing code in tdd", is certainly an alternative one, but is it a good one?

biot 14y ago

My approach to tests is that where you control all dependencies, do tests for core functionality and any tricky parts. After all, you can always hot-fix any breakage since everything is under your control. However, where you have external dependencies (eg: an API that third parties use) you should have fairly robust tests to ensure you are not introducing any regressions.

iow 14y ago

It's true that some tests might get redundant when your functionality changes, but you don't know which part you will throw away at the point you're writing it. Better have that safety net for the part that will need to continue to work after you refactor.

mdkess 14y ago

In my admittedly limited experience, unit tests are way overhyped, especially when things like mocking are brought into the mix. It's easy to end up with a test that is not about correctness, but "did line 1 get called, did line 2 get called, etc.". Then you change the implementation, and 20% of your test cases break. That's not to say that they are valueless, but that I think unit tests should be used pretty sparingly. Where I've found a ton of value has been in writing almost artificial-intelligence driven integration tests. Write a bot that uses your service in way, as fast as possible. Run fifty of these bots simultaneously, and see what happens. Then have some way to validate state at various points (either by tallying the bots actions, or sanity checks). Bugs will come fallout out of the sky. Then, in the future, when you get a bug, the challenge becomes to update the integration test bots behavior so that they (preferably quickly) can reproduce the bug. I mean, I think that this is dependent on the domain of your software, but I think it's a good strategy for many areas.

gknoy 14y ago
If we're only using unit tests to see if line N gets called, I think we're doing it wrong. Instead, we want to use unit tests to tell us if the answer we get is correct -- while exercising line N. e.g.: def test_foo(self): result = get_foo(bar=12) self.assertEquals(result.category, 'FOO') self.assertEquals(result.name, 'my-foo-12') self.assertEquals(result.unit_price, 97.12) This lets you verify that this particular branch (when bar=12) is executed, and that your results are as you expect. If you change some of the underlying calculations, things can break (as you get different answers), but then you at least have a test that lets you ensure that changing the answers is what you want to do. Sometimes, you want to change the way you calculate something and get the same answer, after all.
- mdkess 14y ago
  So the question is, what if bar = 12 means "retrieve foo with ID 12 from the FooService", and we want to make sure that get_foo does something sensible when the server returns an error code. So what do you do? Typically one writes a mocked up service that returns the expected results, pass it in somehow, and write your tests like you did above. So if you have a lot of services, you end up doing what I said - testing lines of code. Here you're testing that FooService gets called. If it also called BarService, you'd be testing that BarService gets called. And so on. But then later, we decide that FooService is no good - we want, nay demand - a FooService2, with a new API. However, get_foo is to behave the same. So what now? Only one choice - update all of the tests to have a mocked up FooService2. As the code base grows, I find this becomes annoying to maintain (although to some extent a necessary evil, of course). The alternative I'm saying is that instead of spending a ton of energy unit testing get_foo, hammer on it and write good state validators. Your unit test shows that get_foo for bar = 12 returns a unit_price of 97.12. Great, but I don't think that's as interesting as building a fake database (so we know what the expected returns are), and then making 50,000 requests per second with random data for 12 hours, and then through runtime assertions and as a post-process validating that the service actually did what it was supposed to.

shubb 14y ago

I'm not necessarily disagreeing with the author, but I think if you are going to 'do agile', especially in a breaky language like C++, you need to do tests. Lots. The current shop I'm at is maintaining a huge code base, parts of which go back 20 years. Because test coverage is so low, there is a real reluctance to refactor. The first thing I did when I started here was to clean up the code, renaming miss-named variables to get it in line with the coding standard, adding autopointers here and there to head off memory leaks. By gosh, I nearly got fired. If you are going to fearlessly edit your codebase, you need to know that regressions are going to be caught. You need automated testing.

azov 14y ago

tl;dr: Tests are overhyped because we once wrote tests that became irrelevant when the thing we were testing changed Really? I mean, tests might be overhyped, but so far it looks like the author draws incredibly general conclusions from some isolated incident. Why did the tests become irrelevant? Is their scoring algorithm now doing something completely different? Do they now have different use cases? Were they hard-coding scores when the actual thing that matters was, say, the ordering of the things they score? > Only test what needs to be tested well, thanks for the helpful advice :) Care to share what, in your opinion, needs to be tested?

jaredsohn 14y ago
>well, thanks for the helpful advice :) Care to share what, in your opinion, needs to be tested? I am not the OP, but I think unit tests are more useful for code that has a lot of edge cases (such as string parsing) and for code which causes the most bugs (as seen in black box testing or integration tests.) Also, some code is just easier to develop if you create a harness that runs it directly instead of having to work your way to the point in the program where it would execute it. If you do this, you might as well turn it into a test.
jamesaguilar 14y ago
I've heard this saying: "Test until fear turns to boredom." Then you need to make sure you get bored at a properly calibrated rate (i.e. not too soon given your business' criticality, not too late given your need to move quickly). For example, if a lot of expensive bugs during final cause analysis are found to be preventable via unit tests, then your calibration is probably tuned to being bored too quickly.
- mmcnickle 14y ago
  That's an excellent saying. It's exactly the line I try to tread with testing. I don't know what to do with the angst of getting the tradeoff wrong though.
- gfodor 14y ago
  This is a great line. I think it goes deeper though, in that after doing this for a while, if it hurts, you are probably doing it wrong. Writing code with too little tests hurts, because you are scared shitless. Writing code with too many tests hurts, because you are performing a religious rite not actually building something useful. The trouble is that you have to do this for a while to get to the point where you can both feel and recognize pain, or the lack thereof.

Skoofoo 14y ago

Behavior/test-driven development needs all the hype it can get. There are so many developers, even entire software development subcommunities (I'm looking at you, video game developers) who haven't written a single test in their lives and don't understand its value. Yes, it is time-consuming and invisible to your customers, but I imagine so is setting up a frame for a house instead of just stacking bricks. The structure, flexibility and peace of mind you get from a comprehensive test suite pays off when you have a 50-brick tall structure to put a roof on.

rhizome31 14y ago
In my experience test-driven development is not time consuming, even for small developments. I don't think we even need to say things like "yes it takes time now but it will pay off later". I have found that it saves time right from the beginning. It seems other people have a similar experience : http://blog.8thlight.com/uncle-bob/2013/03/11/TheFrenziedPanicOfRushing.html http://blog.8thlight.com/uncle-bob/2013/03/11/TheFrenziedPan...

RivieraKid 14y ago

I agree and this is not the only thing that is overhyped right now...

antidaily 14y ago
Also overhyped... trains, The Lumineers, IPAs.

bazzargh 14y ago

Funnily enough what the article describes is almost the perfect case for testing: "We wrote the first scoring algorithm...based on the red-green light." This sounds like it describes some heuristic weighting, which couldn't have been solved by types, but reasonable tests would have shown if the new algorithm weighted some special cases higher or lower than it should. The problem with an heuristic weight though, is that it's an heuristic, judged against other heuristics by taste not proof. The obvious testing approach, ensuring that the score for each test case retains the same order as you tweak the algorithm - is overtesting. You don't care about this total order; you more likely care about ordering of classes of things, rather than ordering within those classes; or simply that 'likely' cases follow an order. Hence, you hit far too many test failures. I'd agree that it's possible to overtest in general, but it's so easy to overtest heuristics that it needs called out as a special case, and it sounds like the problem here.

kosma 14y ago

If the author stopped at the title, he would be one hundred percent correct. It's easy to rant about unit tests. Don't fall into that trap. Be aware that it's a hype, and correct for that fact - and once you do, use your best judgement. In the end, it's you, the engineer, who's shipping the product - not the countless ever-arguing denizens of the internet.

trcollinson 14y ago

I find it interesting that the premise for this entire diatribe is that once some code was written, a pivot was made, and a number of tests then had to be thrown away (or reworked, or re-factored?). Recently I had a code base where we decided that due to a new delightful feature our customers were going to be quite pleased with we would need to switch out our old queuing system for a new one. In doing so well more than half of our huge test suite turned red. This told us two important things 1) that the queuing system touched a lot of areas of code we needed to think about and 2) where in the code the queuing system had touch points. Ultimately we were able to put in the new queuing system, fix the areas that were broken by the change, and have the confidence at the end of the process that we had not broken any of the areas of the code that were previously under test. (This does not mean that our code was bug free of course, only that the areas under test were still working in the prescribed way, but that is a discussion for a different article.) I believe that this would have taken a team of people weeks to do previously. I was confident that the change was ready after only 3 days with 2 developers. I would not trade my tests. There is a cost associated with everything, but I believe tests are the least costly way to get highly confident software built.

embwbam 14y ago

I'll take a crack at my own over-generalization, albeit one I think is more useful. Adding strict types (Typescript!) to your code base gives you way more maintainability than testing. Do it first, then if you still needs tests, write them.

lutorm 14y ago

The importance of testing obviously depends on how severe the consequences of shipping broken code is. If not testing means that some broken UI element on some webpage slips by, that's obviously easier to live with than if not testing means you blow up some $150M piece of hardware.

gfalcao 14y ago

This post is over-hyped: "the cool guy who ships products and applies to cool startups, but they only hire super talented and passionate engineers." --- The text above is just like the blog post: is unfounded and purely based on personal opinions. Having a strong test culture in a company of any size is important for a number of reasons, and all of them are founded by both books and articles, but also by AMAZING products. An engineer who has talent in writing readable and well structured tests is also someone who has a special skill in: creating robust systems, reverse-engineering existing systems and therefore good at being hired by a startup that has NO or very little test culture, which end up being hired by a company that has code that is buggy, error prone, coupled and hard to maintain, so that he can apply all this knowledge to clear crap written by people who doesn't like tests

kasey_junk 14y ago

Tests are not overhyped. They are under-understood. Unit tests are not regression tests are not integration tests. I've encountered tons of teams that don't seem to understand this. Unit tests are more than any other factor a design tool. Like any other design tool (uml, specification, etc), when the design needs to change, you throw them out. If it takes longer to design a system with unit tests than without them 1 of 2 things is true 1) you should not write unit tests 2) you should learn how to write unit tests.

PuercoPop 14y ago

>>> We wrote the first scoring algorithm at Goalee based on the red-green light. Within a couple of weeks we made our first algorithmic change, I think he actually mens an specification change. Tests would help you refactor in case you want to change the algorithm against the same specification. But yeah, is a flawed metric so it shouldn't be put on a pedestal.

spitfire 14y ago

So because his particular tool (or use of a tool) adds inconvenience or overhead means the entire concept is bunk? Don't confuse the thing with the concept. Tests are double checking your work. ("Yup. That condition I said would be true still is!"). The tighter and more lightweight we can get that feedback loop going the better out output and the more confident in our work we'll be.

kscaldef 14y ago

"We wrote the first scoring algorithm at Goalee based on the red-green light. Within a couple of weeks we made our first algorithmic change, and made several quick fix releases to update the scoring methods in the following weeks. By the end of the month, a whole section of our testing suite was almost completely worthless. In a world where cash is king, I wouldn’t mind having those dollars back." What I don't understand about comments like this is that a whole section of your code, both runtime / deliverable code and test code had become worthless. But, you only seem to view the discarded test code as wasted effort. Either the tests have value or they don't. And, if you write tests, and then discard the code they test, you'll likely also discard the tests. But, that doesn't change whether or not the tests had value, nor whether the new tests that you'll write for the new code have value.

dspeyer 14y ago
> What I don't understand about comments like this is that a whole section of your code, both runtime / deliverable code and test code had become worthless Not so. The code demonstrated that the first algorithm wasn't good enough and provided the experience needed to write the second one. The tests (hopefully) made the first algorithm's code maintainable, but it turns out there was no need to maintain it.

mstrem 14y ago

Depending on the context, I fully agree (most stuff here is about web development so in general shipping a broken web page is fine - it wont cause too much damage even when it happens, and it can be fixed quickly). In Safety Critical Software however, I hope tests have been written for every single change, and everything has been checked over and over again. Tests become a lot more valuable and simply having to re write them, may make you think of problems you have introduced in the code.

sippndipp 14y ago

Sounds not very sustainable to me. Ruthless and speedy refactoring is only possible with tests. Better onboarding of new team members is only possible with tests. Updating external libraries is only possible with tests. How many hours did I spent in projects w/o tests figuring out what's going on... things that an easy test could have told me in the first place. In a world where cash (=time) is king, I wouldn’t mind having those dollars back.

doki_pen 14y ago

Like anything in life, you have to make a judgement. There is no silver bullet. Tests are very painful to maintain in all but the most well understood problems. There is a ton of great software we all rely on that doesn't have much testing. Having a ton of people rely and test beta versions of your code is much more valuable then any test suite. Dare to think.

FreeFull 14y ago

It's a good idea to record invariants about your code and your problem. Tests are one way, and there are some other ways. It's a good idea to have the checking of invariants be automatic somehow. Any assumptions about the output of your code that you make but don't record in a checkable form, are something that could potentially be silently broken by a change in the code. And then you're wasting your time debugging rather than solving the problem you're working on.

jamesbritt 14y ago

[I]f any portion of your site changes (pivot, any one?) you’re now in a position of having to rewrite large portions of your tests. And the larger the pivot, the more useless your old testing becomes. I'm under the impression that the tests are written first. If you are writing new code, then first there's a (failing) test, then there is the new code. If you are altering existing code, then pivot or not if something breaks you need to see if the reason is broken code or an inappropriate test, and you fix it before moving on. In any event this should be a gradual process. How common is it that large slabs of code gets changed while ignoring failing tests, or large slabs of new code gets added while skipping the tests? Is it that TDD is, in actual practice, unsustainable, or is the problem with not adhering to TDD? Or something else altogether?

pbiggar 14y ago

Amen! I run https://circleci.com https://circleci.com, where we make all of our money by actually running tests for people, and I still believe this. The goal of your software is to achieve goals, typically business goals. Often the software itself is already one or two steps removed from the value provided to customers. Tests are an extra step removed at least. I wrote a similar piece here: http://blog.circleci.com/testing-is-bullshit/ http://blog.circleci.com/testing-is-bullshit/

Evbn 14y ago
Of you don't care whether your CI tool works, I won't care either, because I won't use it.
- pbiggar 14y ago
  Perhaps I missed where I said I didn't care - where did you see that?

altcognito 14y ago

I find unit tests written after the fact to be pointless. TDD makes a lot of sense because it forces you to think about how you are going to write your code. I like that. If you're writing tests after the fact, integration tests have a much more satisfying finish, and I can't imagine going without them (unless they are painful to implement)

rdtsc 14y ago

Tests for software development == Crimes for political agenda. This is one of those issues you are supposed to lie about. Tests are like crime in politics -- no politician is going to say "I am soft on crime, I think we should reduce sentences." So crazier and crazier laws are created. Mandatory minimum sentences for possessing small amounts of drugs, all kind of craziness. People know it is crazy, but nobody is capable of speaking up and remain standing. Same thing with tests. Nobody can say "fuck it, stop writing tests, the customer will never run this code, or this is too fucking obvious, test stuff that matters". That is considered irresponsible. Everyone is supposed to worry about not having enough tests. Now there a subtle difference and that is when it comes to shipping on a schedule and in many companies tests are ignored -- BUT silently. Everyone loves talking about tests but try and tell you boss you spent 2 weeks on adding tests to your code. They might not like. Try to double the time you promise to do a feature by saying I need to write tests for it. Or if you are given free rein try saying I won't do all the new awesome features, I'll write more tests -- it will be approved probably but everyone in the end will praise the guy who chose to work on and deliver features -- even though they might be completely buggy and unusable. So that is the other side if you will.

StavrosK 14y ago
> Same thing with tests. Nobody can say "fuck it, stop writing tests, the customer will never run this code, or this is too fucking obvious, test stuff that matters". That is considered irresponsible. Everyone is supposed to worry about not having enough tests. I don't know about everyone else, but I definitely worry about having too many tests. For every test I write, I have to weigh how useful the test is versus how likely it is that the thing it is testing will change. If its usefulness is overshadowed by the likelihood that it will be a burden later on, it does not get written.
- rdtsc 14y ago
  Yap. It is very easy to write bad tests -- tests that test too much, tests that test the wrong thing, tests that are too fragile (very timing dependent, they break intermittently and suck up resources in tracking down the cause), tests that are not repeatable (someone inserted random input generation and seeded the generator with the time function).

huntedsnark 14y ago

Another day, another link-bait, faux controversy post about testing on HN. TLDR: Writing bad tests makes for a bad time. Do we need a whole article to cover that? As a side note, all this recent hand wringing over testing reminds me so much of the time when the new breed of version control software was coming out, it was the same kind of accusations about things being "overhyped" and "you can just email a tar file!"

donebizkit 14y ago

I have a long experience in enterprise software and I agree with the premise. There are two kinds of test units: workflow and functional. 1 - Workflow test units are a waste of time because no single test unit stays valid when there is a change. In other words, whenever we added/removed steps in the workflow, 99% of the time we have to change the test unit to fit that new workflow which breaks the concept of "write once, test all the time" concept. In my experience, having proactive developers who test areas around the workflow that they changed is much faster and reliable. 2 - Functional test units are great. They test one function that needs certain parameters and is expected to spit a certain output i.e function to calculate dollar amounts or do any king of mathematical operations. However, these functions tends to stay unchanged during the lifetime of a project. Therefore, the test units are rarely run. From my experience workflow changes/bugs represent 80% of the problems we face in enterprise software. Functional changes/bug are rare and can be detected quickly. This is why I agree with the author premise that unit testing is overhyped.

stcredzero 14y ago

[X] is overhyped != [X] is useless

trustfundbaby 14y ago

In total agreement with the OP. I think it just goes to the way human beings handle original ideas, first they fight them, then they embrace them, then they take them to ridiculous extremes as they try to substitute rules for common sense in applying them. You can see it in politics, religion and almost any really popular area of human endeavor. Testing falls in the same category, I have had interviewers look me in the eye and in all seriousness, declare that developers who don't write tests for their code should be fired, or that their test suites cannot drop below x% of code coverage. Dogma is a horrible thing to afflict software teams, whether it is pair programming, or mandatory code reviews, if there are no exceptions to the rule or situations where you don't have to apply it, its probably a bullshit rule IMO. Me, I like to ship shit, and I like to look at tests as a way to help me ship shit faster, because the less time I spend fixing regressions the more time I can spend actually getting more code/features out that door. So my only metric for writing tests is this ... "is this going to help (me or a team member) not break something months from now, when I change code somewhere else". I honestly don't care about anything else.

Evbn 14y ago
Tests certainly interfere with your ability to ship shit.

teek 14y ago

My general rule: If you are planning on some sort of reuse (API, library, protocol, etc) a test spec is a very good idea if only to hash out practical use of the interface and identify caveats in the interface spec. Once released, the test spec is great for answering the question "did I accidentally break something that used to work?" thus vastly speeding up the bug-fix process for underlying components or widely used components. If you are writing an end-user app, or a service that will be used non-programmatically (such as a GUI or web app), then a test spec is not required and probably a waste of time as things will likely change too quickly.

GhotiFish 14y ago

I don't know what to do with tests, to be honest. I am a very inexperienced developer, and the general sentiment is "knowing what to test is a learned skill" damn. I could use some opinions actually. The current workplace arguments about what we should or shouldn't test go like this: "We write tests to establish specifications, then we write code to meet those specifications. We should have tests covering every aspect of the system as a way to assert that things are meeting the designed specs." "but our model objects have allot of tedious detail. Why should we write tests about what the name field can and can't accept? It's a max length of 32, that's it. Asserting that the constraint fails is beyond tedious and a waste of our time. Most of the code we'd be testing is auto-generated. Are we running tests on our framework or our code?" "Some of this does seem like a waste of time, but what about when someone changes something as part of the development cycle? eg. Someone needs to remove validation from the name field to test other parts of the systems behavior with an overly large name. If we had a complete testing system in place it would alert us that the model isn't meeting the spec. We've already had instance where temporary code to bypass authorization was accidentally pushed to the develop branch." On one end of the spectrum there is this base level of tests that are actually REALLY useful when developing features. Doing simple testing has helped me catch off by one errors and other kinds of bugs, I can stand by my code confidently. These kinds of ad-hoc in development tests don't cover everything, but they're useful. They're like a substitute for step through debugging to verify code. On the other, I don't know how big of an issue bug regressions will be, we've heard other developers suffering for lack of a solid testing base to detect problems. Without a full battery of tests asserting everything we know about the spec, there's no way we will catch many regressions. HN what do you think?

columbo 14y ago
> I don't know what to do with tests, to be honest. ... HN what do you think? Well, since you asked! The most valuable tests in any system is functional tests. Unfortunately most tests are unit tests that give little value. Here's something that I encountered recently in the wild. Imagine a situation like this, a calculation process with three distint black boxes of code. Block A - Pull data from 3rd party source Block B - Distribute data to individual nodes (multi-threaded environment) Block C - Individual nodes processing data Each one of these blocks is around fifty thousand lines of code, each one of these blocks has hundreds if not thousands of unit tests. Every line of code is tested by a unit test. Yet there is a very real and dangerous bug in the system. Why? Because the object passed from B-C has a transient variable and when it is serialized and deserialized that value is lost and reverted to the default value. Now most places leave this value as the default, so it's only one out of a thousand customers that have this problem, but when they do it's a huge problem. Functional tests is the process of saying "I put meat/spices on this side and sausage comes out the other" it doesn't try to determine the RPM of the grinder, the number of bolts, the rotation of the gears, or the type of metal used to build the sausage-maker. It is simply the process of determining that meat becomes sausage. Functional tests have high value and are absolutely worth knowing. In most CRUD applications Unit Tests generally wind up testing that the universe still exists as expected (true is still equal to true, zero is not one, and a null value is not the same a string, oh happy days the system works!). That's not always the case. If you are working on something that is more advanced, say something that should measure the chance a hurricane might knock down a given 3d model you'll wind up having a huge stack of unit tests, and these will be very valuable. More often than not you'll know they are valuable as they'll be spent proving the science behind the application, and not simply that the application 'does something'.

ryan-allen 14y ago

I think the hype of tests is driven by agencies that charge by the hour. One thing I have noticed is that on an existing system especially, you can put a very inexperienced developer down and instruct them to 'write tests for untested functionality', and do so but produce neligible value. The same thing with green fields development. I've seen steaming piles of shit with huge test suites. Absolutely zero insight into the problem. No craftmanship at all, nothing interesting about the application. But a set of tests. It's like the suite is proof enough that there was a job well done. I fear that a lot of development is devolving into nothing more than superstition and hype, backed up by agencies that like to bill a lot and amateurs who need a justification for their timelines and ineptitude.

pesenti 14y ago

Seeing how many people agree with the post I am left wondering who the commenters on HN are. With a large code base and actual customers, comprehensive unit testing is by far the most cost effective way to create and maintain software. This is not being dogmatique, it's the shared experience of the majority of established software engineering firms. In the case of my company, we experienced so much growing pains that our productivity almost came down to zero before we switched to TDD. Many developers in our group were skeptics (including me) but today you won't find one of us arguing for less testing. The kind of large pivot that the author refers to is only possible when you don't have established customers and you have a minimal product. You may as well call it prototyping. And prototyping with or without tests is indeed more a matter of taste than effectiveness.

laureny 14y ago

His beef is more with unit tests than functional tests, and I can't disagree with that. Functional tests code that will directly impact your users while unit tests are merely there to make the developer's life more comfortable.

dvt 14y ago

Even though I can sympathize with the tone of the OP, I think his argument cannot be generalized. Consider the following scenarios: A one-person engineering team. This is some 20-something working on a start-up in his parents' garage or the 40-something code hobbyist out there that likes writing fun little apps every now and then. I would argue that there is virtually no need for testing unless you're either bad at writing code or inexperienced. I've written dozens of applications (in various languages) and once you get into the swing of things, you don't need TDD to actually "test." Debugging/edge cases/etc. just becomes a natural process of writing code. A two-person engineering team. This is a secondary stage in every start-up -- sometimes, start-ups even are founded by two engineers. Here is where TDD starts being important. I may be an expert coder, but if my partner isn't that great (or if our code doesn't work well together), not testing can be a huge nightmare. But the impact is still relatively small. Bugs are easy to spot as both engineers are (or at least should be) incredibly familiar with the entire codebase. Three-person to 10-person engineering team. Here is where things get really tricky and TDD becomes integral to meeting deadlines and actually, you know, putting out products. You've got Jim working on some financial stuff while Bob is optimizing some JPA code. At the same time, Susan is implementing a couple of new features. Having a test suite ensures that behavior will remain predictable given non-centralized iterations. Without TDD, large teams would be fixing bugs more than they would be writing code (most teams probably do that anyway -- not that it's a good thing). 10+ people; open source projects, etc. When you have more than 10 people working on a project, I think that TDD is simply a necessity. There are simply too many moving parts. Without a cohesive test suite, having this many people mess around with code will undoubtedly break stuff. Ironically, I think that small teams put too much emphasis on TDD, whereas teams in larger companies (AT&T and IBM, for example) put too little.

camus 14y ago

tests should be fast and easy to write , that's why any library should be bundled with things that make it easy ( null classes , mocked classes , ect...).

digitalWestie 14y ago

Can we please stop referring to 'testing' as if it's the same thing as TDD?

pkolaczk 14y ago

No amount of tests can help you if don't know how to solve the problem. And the better you know how to solve the problem the less tests you really need. ;) Old, but quite relevant: http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-solvers.html http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-s... Some sane amount of testing is good. However, I'm not very convinced writing tests first is a good idea. I saw a few programmers practicing this and when writing code they often concentrated too much on just passing the damn test-cases instead of really solving the problem in a generic way. Their code looked like a set of if-then cases patched one over another. Therefore, if they missed an important case in their testing suite, they had 99% chances for a bug. Once I ended up writing a separate, more generic implementation to validate the test suite. And it turned out there were a few missed/wrong cases in the test suite.

dwhelan 14y ago

There seems to be a sense that writing tests slows you down. And that is understandable. However, I find that tests helps me move in small steps and provide a feedback loop that helps me stay in a groove moving towards the finish line. And they help me move faster. I think we as developers need to continually learn how to write better tests that provide deeper value and allow us to move more quickly. I find that when testing is tough it is often that the underlying design is deficient. And tests shine a light on code and design smells. Discarding tests could be valuable if the tests have outlived their value. It might also be the case that excessive test maintenance is telling you something about your production code.

TeeWEE 14y ago

Man I like this article so much. I worked at a company that aimed at 90% test coverage. Without taking into account how important the piece of code under test was. I think the 80/20 rule applies here. Only 20% of your code does 80% of the work. Focus on writing tests for those 20% then your time is much better spent.

Paddy3118 14y ago

Without due scientific process to back up TDD we are left with such arguments having equal validity as TDD. Without a scientific proof of TDD being superior then we are left with mere opinions on both sides. NAARP! http://paddy3118.blogspot.co.uk/2013/02/no-attempt-at-rigorous-proof.html http://paddy3118.blogspot.co.uk/2013/02/no-attempt-at-rigoro...