15 ms·

But we don't write tests to check if our code works. We write tests to be able to change it in the future with certain degree of confidence that we don't break

But we don't write tests to check if our code works. We write tests to be able to change it in the future with certain degree of confidence that we don't break anything and - if so - what exactly. There are other techniques which can give similar confidence, but tests are the easiest one.

smallnamespace 10y ago
'Code that works' doesn't necessarily mean 'code that only works today'. I'm pretty sure maintainability is implied in his statement.
- alkonaut 10y ago
  If maintainability is part of "works" and tests are a necessity for maintainability (they are), then the quote becomes prettu weird. Substituting [code and tests] for [code that works] gives > I get paid for [code and tests] not for tests.
  - smallnamespace 10y ago
    The discussion was around what the right level of testing is. > I get paid for [code that works and is maintainable], not [more tests than are strictly necessary to achieve that goal]
    - alkonaut 10y ago
      Yeah, that's the reasonable interpretation (which is also backed up by the full article). As usual the quote is taken out of context and/or put into a headline that's deliberately much more inflammatory than the actual article.
- pif 10y ago
  > I'm pretty sure maintainability is implied in his statement. I don't know this guy, so I can't speak for him in particular. But, in general, I wouldn't be surprised if the opposite was true: too many so-called software developers give "shipping" too much importance, leaving none for any other aspect of the job. Shipping is a feature, not the feature.
  - imarg 10y ago
    If by "I don't know this guy" you mean Kent Beck then you should google him[1]. This is not some random guy, he is like the "father" of TDD. And that is why the guy that wrote the article thought it noteworthy to mention his quote. If it came from someone else it would not be that important to make such a fuss about it. But when it comes from Kent Beck then it is worthy at least some discussion. [1] https://en.wikipedia.org/wiki/Kent_Beck https://en.wikipedia.org/wiki/Kent_Beck
  - squeaky-clean 10y ago
    > I don't know this guy, so I can't speak for him in particular. Literally the second sentence in the article. > Kent Beck, respected authority, creator of Extreme Programming, TDD and writter of several great reference books, mainly at the great Addison-Wesley edition
  - switchbak 10y ago
    Kent Beck brought developer testing to prominence in the early 2000's with his books on XP and particularly TDD. Back in those days, there was a backlash against "big design up front", and very little respect (in general) for testing as a practice. Unit testing in it's modern form was reasonably rare. After this Agile/TDD stuff caught on, many people ended up over-testing things. This is a pretty typical thing to do when you're learning about how much testing is sufficient. I've definitely done a good amount of this myself. It can take a good deal of experience to know where to draw the testing lines in particular contexts. I think this blog post points at this specifically - that we should write high-value tests, and just enough of them. We also use feedback over the long-term to have heuristics of where we tend to have recurrent issues, so we can test a bit more in those areas. Far from being focussed on "only shipping", he's underlining the fact that "just enough well-written tests" support working software - and that should be our focus, instead of thinking our job is to "write more tests" (or focus on test coverage, etc).
fortytw2 10y ago
Removing implicit side-effects and global state from code and having strong, static types can go very far in allowing this, without much overhead. Unfortunately, there isn't a really great language that enforces working like this, while being simple enough to push onto a big team :/ (if there is, please let me know)
- bpicolo 10y ago
  F# is probably a pretty decent candidate for this (it's next on my list of languages to really take a critical look/attempt at). I've done a lot less statically-typed code than dynamic (mostly Python) in the last few years, but I was playing with Unity3d recently and had a chance to write a good deal of C# with Visual Studio. I hit a hairy problem some time along and had to do a big refactor to support a new feature. I deleted a single line of code, followed an error trail for 10 minutes, and suddenly everything was just done. It was a pretty interesting moment for me. I realized that statically-typed languages can really have the potential to be as or more productive than dynamically typed languages paired with a good enough IDE. (And Visual Studio with C# is about the best pairing you can get). I've had to reconsider my thoughts on these things a bit. Sure, there are things that statically typed languages can make much harder to test or work with. (Want to stub an external provider? Okay, you're going to need an extra interface, then you'll need to create a new stub version that implements that. Want to read/write pretty-arbitrary JSON? Good luck). But there are other places where you get huge wins by bugs just disappearing by the boatload. I'm still not on board with heavy OO/inheritance, and love the pattern of simpler struct-style constructs with just functions in functional programming, but the static typing can give a lot of wins. I think something that gives an inherent advantage to OO languages in IDEs is that SomeThing.<tab autocomplete> makes a lot of sense and is easy to compute! I can take an object and know what I can do with it at a glance. I haven't seen a functional language with enough structure to support that simple feature yet (though maybe I'm just not looking hard enough). This is really where statically typed languages can make the most of an IDE. For some reason, that's the big thing I think of when I'm thinking about the downsides of functional programs I'm working with. The editors just seem to help a lot less (though I haven't written any FP professionally-speaking, so have less experience in general with tooling). F# using Records with member methods looks like it might be able to get that sort of benefit though, I'll need to try that. It looks like they're just pure functions declared on immutable structs, which I think is the perfect middle ground. A lot of object-oriented languages have taken tips from functional languages lately (map/reduce/filter is the new hottie), but I think there's a lot of benefit still to get in the opposite direction.
Arnt 10y ago
That's one reason to write tests. Another is that some code reviewer asks for more tests. A third reason to spend time on tests is that they're required to maintain the same standard as the shipped code, even though they're only run in the presence of the developers, and their breaking only affects the developers, not any customers. People forget the ultimate reason for our work oh so often.
- agentultra 10y ago
  I don't have that kind of hubris. I value reliability, repeatability, and consistency. And so do my customers by tell of the issue reports we get. They're most upset when something goes wrong. I write tests, many tests, when I'm working in a dynamically typed programming language. I write tests even when I'm working in a soundly typed language. The only difference is that in soundly typed languages the type system guarantees many properties for me so I don't test for those. Personally I like to write tests first but I don't believe that gives me any productivity benefits. It's just the way I think. > People forget the ultimate reason for our work oh so often. Tests are important because reliable code is important. The customers are important but so is the business. It costs quite a lot of money to support error-prone, poorly designed software. Tests aren't a silver bullet but they are a tool to alleviate the problem.
  - Arnt 10y ago
    I've heard words much like those many, many times, but always in general or specific to a context where it didn't apply. For example when I changed some code for which tests didn't exist, so I tested what I changed and wrote some extra tests while I was at it, and it was blocked in code review because my extra tests didn't report failure in any detail. What I did said "x failed" if a test failed, no details. The reviewer said much the same as you did now to justify that additional reporting was absolutely required. It's a fine sentiment when it actually applies, and I wish it weren't applied quite to often to justify YAGNI and other rubbish.
    - switchbak 10y ago
      Can you say specifically what you object to in his words above? I'm failing to connect what he said with your complaint. The advice you received with regards to defect locality sounds reasonable - tests that don't give much in the way of isolation can cost a lot of developer time to hone in on. It's hard to say, not knowing the exact details however. I also find it hard to reconcile the idea that "tests are an important tool for writing good software" with "justifying YAGNI". How would an "openness for developer testing" justify an attitude of "not writing things you don't need"? Those two concepts sound almost entirely orthogonal.
Alex3917 10y ago
> But we don't write tests to check if our code works. I write tests to check if my code works. And tests that document how the code is supposed to work currently are usually enough to prevent code from breaking in the future. Anything related to privacy or security should be fully tested. But for the typical startup, I'd posit that beyond that test coverage should be more closely related to the number of users and level of usage rather than to the amount of code.
- e40 10y ago
  The number of times my tests have 1) found bugs in my code, and 2) exposed problems with the API I was about to publish... uncountable.
- hannob 10y ago
  > Anything related to privacy or security should be fully tested. How often do you write code that's not related to privacy or security? As soon as you connect something to the Internet it's related to privacy and security. The only situation where privacy and security don't matter a whole lot is if your code runs on airgapped devices with very limited tasks.
  - EpicEng 10y ago
    I rarely write code that connects to anything, so almost never? There's a lot of code that's not written by web devs.
    - hannob 10y ago
      > I rarely write code that connects to anything, so almost never? Sounds hard to believe. What kind of code would that be? > There's a lot of code that's not written by web devs. There's also a lot more than the web that has some form of connectivity with the Internet (even if it's not directly connected it may still parse data that comes from untrusted sources). There is a widespread belief among many that "security is important, but doesn't matter for me". The most extreme example is obviously IoT ("Who would want to hack my coffeemaker?"), but there's a lot more. The unfortunate truth is: There is hardly any code these days that is not security relevant.
Touche 10y ago
The quote is not anti-test. You should read the article.
terryf 10y ago
The first part of your comment sounds really odd to me. I definitely write tests to check if my code works, among other things. Why do you think that is the wrong approach?
- cheez 10y ago
  I spent some time working for one of the biggest names in trading technology and one of the things I learned is that if your code doesn't work on more or less the first try, your design is probably wrong. The implications of that lesson are profound. When something doesn't work as expected, I now check my design and not the code.
  - pc86 10y ago
    > if your code doesn't work on more or less the first try, your design is probably wrong Or you're tired. Or you're just not as focused as you could be. Or you have a deadline. Or you're trying something new. Or you're not fluent in the language yet. Or you're fluent in the language but not the framework. Or you're fluent in the language and framework but not the design pattern (if you use those). Or a whole host of other things. I'm not saying spot-checking the design isn't a good idea, but saying that it's the design more often than the code just doesn't match up with my experience.
    - cheez 10y ago
      The entire idea is to increase the probability that you are correct. So if you're coding tired, you're decreasing that probability. If you're using a framework you're not familiar with, you're decreasing that probability. This is why I mentioned that it was in trading technology: they only get paid to be right.
  - lordnacho 10y ago
    How do you define "the first try"? Surely you're bound to have various trivial issues?
    - 10y ago
andrewstuart2 10y ago
And we write tests to exercise our own assumptions about how the system will be used. It's way too often I write some tests for my code, I quickly arrive at use cases and failure cases that I hadn't anticipated while writing for the optimal use case. Reality, on the other hand, is usually suboptimal.
Quarrelsome 10y ago
yes but if you follow that too strongly you end up having to rework tests _every_ time you change something which is absurd. You should be testing input/output and results as opposed to testing how the internal gubbins works. That's the line we have to carefully tread when making a test. The test shouldn't force the item to behave in directly the way it expects; more that the I/O is correct.
- gr3yh47 10y ago
  > yes but if you follow that too strongly you end up having to rework tests _every_ time you change something which is absurd. it depends how brittle your tests are. You can write tests that make sure internal stuff works at a unit level without them being so brittle to your point testing I/O or behavior is the way to accomplish this, but it can still be done at the level of internal functions/methods
- demonshalo 10y ago
  this is essentially the argument for functional tests over unit tests and while I generally agree, I think a mixture of the two is important. Unit tests should be used for extremely small and isolated mission critical objects while functional tests should generally cover the entirety of the I/O chain. That's how I do it at least and it works extremely well for a fraction of the cost!
  - sethammons 10y ago
    How do you test error conditions? In my experience, you are typically not mocking hardly anything (ideally nothing) with functional tests and some error cases require mocking. I find unit tests helpful in this arena.
    - crdoconnor 10y ago
      I mock things all the time with functional tests. It makes it easier to reliably test your code's response to unusual conditions (e.g. error conditions) and it eliminates a source of brittleness in the test (you can write functional tests that hit the real paypal API and run them every day but every 2nd Friday they will fail because paypal is shit at keeping their servers up). This is in no way a benefit unique to unit tests.
raldi 10y ago
I agree with your assessment that the primary reason we write unit tests is to be able to quickly make changes in the future without fear of breaking something: You do some extra work today so that you can save a tiny bit of work (and worry) tomorrow, and over time that accumulated future benefit outweighs the penalty paid today. However, many in the industry forget that this is the underlying reasoning, as seen just two days ago here on this site. Read through the top-rated comments on this post: https://news.ycombinator.com/item?id=13119138 https://news.ycombinator.com/item?id=13119138 They describe an emergency situation where a single "3" needed to be changed to "4" ASAP or people would lose their jobs, and everyone's applauding the gatekeepers who insisted on significant refactoring and the creation of additional tests before the change could be approved. I agree with those who say those improvements should maybe have been demanded immediately after the fire was out, but those who would have delayed the firefighting out of blind allegiance to the rules seemed, to me, to have forgotten that the rules are there to serve the programmers (particularly, their ability to quickly ship working code), and not the other way around. A rule that's failing to do that should be changed or ignored.
- wnevets 10y ago
  Everything in moderation.
  - codeduck 10y ago
    including moderation.
  - nurettin 10y ago
    That's rather extreme.
  - matthewmorgan 10y ago
    I prefer a slightly different translation of that Ancient Greek saying: nothing too much
  - 0xdeadbeefbabe 10y ago
    For a moderate amount of everything.
gaius 10y ago
We write tests to be able to change it in the future with certain degree of confidence that we don't break anything and - if so - what exactly You don't want tests for that - you want a type system and a static analyzer.
- cm2187 10y ago
  But only tests will tell you you are calculating something correctly.
- digi_owl 10y ago
  But thats anathema to agile, yo. /s
- raldi 10y ago
  How does that confirm to you that, say, your ratelimiter code is correctly processing the first three requests in a given minute, rejecting the fourth, and approving a fifth once another minute has passed?
- jon-wood 10y ago
  A type system and static analysis certainly allow you to stop writing a certain category of test as you no longer need to assert that a method being passed an object of the wrong type copes with that. It doesn't completely relieve of you the need to test though as a type system won't assert that the code you're writing actually fulfils the businesses requirement that nothing can be ordered for same day shipping after 11am.
  - gaius 10y ago
    You absolutely can do that with the type system, use a constrained subtype.
    - jon-wood 10y ago
      You've got my interest piqued now. How does that work in practice? And do you not still need to test that it complies with the actual business rules at some point?
fsloth 10y ago
"But we don't write tests to check if our code works." If the input data is complex enough to drive a non-trivial branching logic in our code - yes we do. Well, at least I do. The bonus is that it provides the fixed point for further changes at the same time.
misja111 10y ago
There are a couple of reasons why you should/ could write tests: - for catching regressions in the future - to verify that your code works. How else would you know that your backend service reacts properly on some edge condition - as documentation - as a kind of quality mark, for instance to be able to pass a code review or when writing code for an external party Unfortunately the last reason is also the most useless and still it is the one that seems to be the main motivation for many big enterprise developers.
- TallGuyShort 10y ago
  >> - as documentation Thank you for including this - it's underappreciated. So many times recently I've needed to write code against some poorly documented API (not always due to lack of effort, some things are hard to document well in prose / JavaDocs, or just due to constant change), but I took one look at the unit tests and it all made sense - and I knew it was up to date for the latest work.
  - Freak_NL 10y ago
    Tests are great for dyslexic colleagues as well. A good clear concise unit test is often easier to grasp than the best of API documentation (e.g., JavaDoc in Java).
    - TallGuyShort 10y ago
      Good point - I'd bet it applies to autism and such too.
      - euyyn 10y ago
        I think it applies to everybody. I think it was Terence Tao who had a blog post claiming (and showing) that teaching by example first is the most effective.
- mannykannot 10y ago
  I know this is not formally an ordered list, but it looks cart-before-the-horse-ish to have catching future regressions before verifying that it works in the first place. (This comment is effectively a reply to the OC's claim that we don't test to verify.)
EGreg 10y ago
So then why don't you just write the tests BEFORE you are about to refactor something or change some interface? Why write them at "launch time" of the feature?
- st3v3r 10y ago
  Because if you haven't written them, odds are the code is not particularly well suited to be testable. Side effects out the wazoo, tight coupling, etc.
jsight 10y ago
I have seen codebases where the developers diligently insured that every getter and setter method had a test case. They had great test coverage, but not particularly reliable outputs. OTOH, projects with excellent end-to-end tests tend to have lower coverage but better regression management. I think this is the point of the article?
- eveningcoffee 10y ago
  I think that this is one of the points. I think that full test coverage is nor sufficient or necessary for the working software. What matters is the sufficient covering of the edge cases. I think that an additional point to keep in mind is that most probably Kent Beck talked mostly from the point of statically typed languages.
randomdata 10y ago
Technically, we write tests to document what the code does. We could use any language, including english, to do that. However, using a programming language carries the side benefit of being able to allow automatic verification that the documentation is at least in sync with the code (although not necessarily accurate with respect to the project goals).
- sidlls 10y ago
  And then who or what documents the tests? Test code is code. That's important and all too often that fact is sort of elided. "We" write comments and use "plain" language documentation, or formatted but otherwise plain language for documentation generators to parse to document our code, and that includes "what the code does." I'd argue that if your test code is what you're relying on to "document what the code does" then you are probably (almost certainly) over-testing, testing the wrong things, or some combination. Oh, and also using test code incorrectly (as documentation).
wvenable 10y ago
> We write tests to be able to change it in the future Which works great as long as your changes are shallow. If your changes aren't shallow then you have to change your tests as well and that defeats the purpose. Automated testing is good for freezing an interface and it's behavior; allowing you to change implementation while maintaining the same outputs. Lots of technology requires this: Networked API endpoints, libraries, etc. But most change I encounter is from changes in requirements that necessarily requires reworking and re-arranging code that won't be compatible with the test suite.
- Jtsummers 10y ago
  Different types of tests. You can have tests of the whole system which should always pass. Then there's unit tests, which may have to be removed/updated (though again, this depends on your scale of units). Some you expect to fail after a change to requirements get incorporated. Others you expect to pass, and you react accordingly.
- olavk 10y ago
  If the code is designed following the single responsibility principle, and tests are written to test behavior rather than implementation details, then each requirements change should only affect the tests which specifically test for the behavior which changes.
  - wvenable 10y ago
    That's still a very narrow view of changes; the belief that you can isolate every part and that requirements changes would then only affect those isolated parts seems is pure fantasy. I've had to completely redesign entire subsystems; break down components into different pieces; move code between different layers; etc. Honestly as a manager nothing bothers me more than developers who try and patch complex changes into existing systems without re-thinking how it affects everything else. I have to re-factor a project that's a total mess because code was added but not removed or changed over the course of some very big requirements changes. I'm sure all the tests pass but it's impossible to follow now. The problem with unit tests is it adds an extra layer of friction on making changes that benefit the product. You are actively discouraged from changing your design from your initial assumptions! This change friction can be seen as a benefit if you need all your interfaces to be stable (like with a library). But it's a trade off and it's not appropriate everywhere.
gshulegaard 10y ago
This is the right answer. I will also point out that as a result of this property collaboration becomes easier.
vikiomega9 10y ago
N00b question, what's another technique? I mean if tests are assertions on assumptions is there something better?
0xdeadbeefbabe 10y ago
That only works if you run the tests often. Writing them is less important than running bits of your program often enough to discover how it really works.
mannykannot 10y ago
> But we don't write tests to check if our code works. So what methods do you use for that purpose?
partycoder 10y ago
Tests can be seen as a form of executable documentation, or a executable specification, a sort of self-enforcing contract.
hinkley 10y ago
When I start thinking of some of my code as safety equipment, and some like shop tools, a lot of my decision processes become more straightforward. My tests and practices are, occasionally, like a five point harness in a race car. Without it anchoring the driver in front of the wheel, they could never ever drive that fast with any safety at all. This one comes out as a counter to the 'straightjacket' argument some people like when the tools won't let them just write shit code that the rest of us have to babysit while they run off somewhere else to write more shit code. Which we are supposed to be grateful for... I digress. But most of the time my tests are more like smoke detectors. The ones that go off for no reason get replaced or removed. The ones that go off after I can see fire and smell smoke? Why am I paying for upkeep on those exactly? What's the value?
Steeeve 10y ago
Absolutely. The article itself (it's a short one) talks about doing just enough tests to have confidence in a product, and I think that's fantastic for early phase development. But when changes can come from anywhere (lots of developers potentially making changes) and the need for changes can be urgent, there's a great deal of value in complete code coverage. Not every developer thinks the same or has the same tendency for errors. I would caution people against thinking about tests as personal verification. With any luck, you'll be handing off that code base to someone else in time.
mrmrcoleman 10y ago
This is entirely subjective as it comes down to Kent's sense of what needs to be tested, which is arguably a better 'sense' than that of many other programmers. Do you allow your juniors to use their sense? No, you make them hit 100% coverage until they start to learn. As Paul McCartney said: "Learn the rules like a pro, so you can break them like an artist."
zebraflask 10y ago
That's not a bad perspective, but the counterpoint that I have seen several times is that the tests can tend to take on a split quality where some will insist that nothing can be changed if a test has to be changed along with it - and then the other camp steps in and doesn't want to put in the effort for new tests, so the existing ones get removed. Testing makes a lot of sense if you're doing data munging, but for front end or other kinds of code that are almost defined by their ability to create side effects, it's usually a waste of time. In my humble.

smallnamespace 10y ago

'Code that works' doesn't necessarily mean 'code that only works today'. I'm pretty sure maintainability is implied in his statement.

alkonaut 10y ago
If maintainability is part of "works" and tests are a necessity for maintainability (they are), then the quote becomes prettu weird. Substituting [code and tests] for [code that works] gives > I get paid for [code and tests] not for tests.
- smallnamespace 10y ago
  The discussion was around what the right level of testing is. > I get paid for [code that works and is maintainable], not [more tests than are strictly necessary to achieve that goal]
  - alkonaut 10y ago
    Yeah, that's the reasonable interpretation (which is also backed up by the full article). As usual the quote is taken out of context and/or put into a headline that's deliberately much more inflammatory than the actual article.
pif 10y ago
> I'm pretty sure maintainability is implied in his statement. I don't know this guy, so I can't speak for him in particular. But, in general, I wouldn't be surprised if the opposite was true: too many so-called software developers give "shipping" too much importance, leaving none for any other aspect of the job. Shipping is a feature, not the feature.
- imarg 10y ago
  If by "I don't know this guy" you mean Kent Beck then you should google him[1]. This is not some random guy, he is like the "father" of TDD. And that is why the guy that wrote the article thought it noteworthy to mention his quote. If it came from someone else it would not be that important to make such a fuss about it. But when it comes from Kent Beck then it is worthy at least some discussion. [1] https://en.wikipedia.org/wiki/Kent_Beck https://en.wikipedia.org/wiki/Kent_Beck
- squeaky-clean 10y ago
  > I don't know this guy, so I can't speak for him in particular. Literally the second sentence in the article. > Kent Beck, respected authority, creator of Extreme Programming, TDD and writter of several great reference books, mainly at the great Addison-Wesley edition
- switchbak 10y ago
  Kent Beck brought developer testing to prominence in the early 2000's with his books on XP and particularly TDD. Back in those days, there was a backlash against "big design up front", and very little respect (in general) for testing as a practice. Unit testing in it's modern form was reasonably rare. After this Agile/TDD stuff caught on, many people ended up over-testing things. This is a pretty typical thing to do when you're learning about how much testing is sufficient. I've definitely done a good amount of this myself. It can take a good deal of experience to know where to draw the testing lines in particular contexts. I think this blog post points at this specifically - that we should write high-value tests, and just enough of them. We also use feedback over the long-term to have heuristics of where we tend to have recurrent issues, so we can test a bit more in those areas. Far from being focussed on "only shipping", he's underlining the fact that "just enough well-written tests" support working software - and that should be our focus, instead of thinking our job is to "write more tests" (or focus on test coverage, etc).

fortytw2 10y ago

Removing implicit side-effects and global state from code and having strong, static types can go very far in allowing this, without much overhead. Unfortunately, there isn't a really great language that enforces working like this, while being simple enough to push onto a big team :/ (if there is, please let me know)

bpicolo 10y ago
F# is probably a pretty decent candidate for this (it's next on my list of languages to really take a critical look/attempt at). I've done a lot less statically-typed code than dynamic (mostly Python) in the last few years, but I was playing with Unity3d recently and had a chance to write a good deal of C# with Visual Studio. I hit a hairy problem some time along and had to do a big refactor to support a new feature. I deleted a single line of code, followed an error trail for 10 minutes, and suddenly everything was just done. It was a pretty interesting moment for me. I realized that statically-typed languages can really have the potential to be as or more productive than dynamically typed languages paired with a good enough IDE. (And Visual Studio with C# is about the best pairing you can get). I've had to reconsider my thoughts on these things a bit. Sure, there are things that statically typed languages can make much harder to test or work with. (Want to stub an external provider? Okay, you're going to need an extra interface, then you'll need to create a new stub version that implements that. Want to read/write pretty-arbitrary JSON? Good luck). But there are other places where you get huge wins by bugs just disappearing by the boatload. I'm still not on board with heavy OO/inheritance, and love the pattern of simpler struct-style constructs with just functions in functional programming, but the static typing can give a lot of wins. I think something that gives an inherent advantage to OO languages in IDEs is that SomeThing.<tab autocomplete> makes a lot of sense and is easy to compute! I can take an object and know what I can do with it at a glance. I haven't seen a functional language with enough structure to support that simple feature yet (though maybe I'm just not looking hard enough). This is really where statically typed languages can make the most of an IDE. For some reason, that's the big thing I think of when I'm thinking about the downsides of functional programs I'm working with. The editors just seem to help a lot less (though I haven't written any FP professionally-speaking, so have less experience in general with tooling). F# using Records with member methods looks like it might be able to get that sort of benefit though, I'll need to try that. It looks like they're just pure functions declared on immutable structs, which I think is the perfect middle ground. A lot of object-oriented languages have taken tips from functional languages lately (map/reduce/filter is the new hottie), but I think there's a lot of benefit still to get in the opposite direction.

Arnt 10y ago

That's one reason to write tests. Another is that some code reviewer asks for more tests. A third reason to spend time on tests is that they're required to maintain the same standard as the shipped code, even though they're only run in the presence of the developers, and their breaking only affects the developers, not any customers. People forget the ultimate reason for our work oh so often.

agentultra 10y ago
I don't have that kind of hubris. I value reliability, repeatability, and consistency. And so do my customers by tell of the issue reports we get. They're most upset when something goes wrong. I write tests, many tests, when I'm working in a dynamically typed programming language. I write tests even when I'm working in a soundly typed language. The only difference is that in soundly typed languages the type system guarantees many properties for me so I don't test for those. Personally I like to write tests first but I don't believe that gives me any productivity benefits. It's just the way I think. > People forget the ultimate reason for our work oh so often. Tests are important because reliable code is important. The customers are important but so is the business. It costs quite a lot of money to support error-prone, poorly designed software. Tests aren't a silver bullet but they are a tool to alleviate the problem.
- Arnt 10y ago
  I've heard words much like those many, many times, but always in general or specific to a context where it didn't apply. For example when I changed some code for which tests didn't exist, so I tested what I changed and wrote some extra tests while I was at it, and it was blocked in code review because my extra tests didn't report failure in any detail. What I did said "x failed" if a test failed, no details. The reviewer said much the same as you did now to justify that additional reporting was absolutely required. It's a fine sentiment when it actually applies, and I wish it weren't applied quite to often to justify YAGNI and other rubbish.
  - switchbak 10y ago
    Can you say specifically what you object to in his words above? I'm failing to connect what he said with your complaint. The advice you received with regards to defect locality sounds reasonable - tests that don't give much in the way of isolation can cost a lot of developer time to hone in on. It's hard to say, not knowing the exact details however. I also find it hard to reconcile the idea that "tests are an important tool for writing good software" with "justifying YAGNI". How would an "openness for developer testing" justify an attitude of "not writing things you don't need"? Those two concepts sound almost entirely orthogonal.

Alex3917 10y ago

> But we don't write tests to check if our code works. I write tests to check if my code works. And tests that document how the code is supposed to work currently are usually enough to prevent code from breaking in the future. Anything related to privacy or security should be fully tested. But for the typical startup, I'd posit that beyond that test coverage should be more closely related to the number of users and level of usage rather than to the amount of code.

e40 10y ago
The number of times my tests have 1) found bugs in my code, and 2) exposed problems with the API I was about to publish... uncountable.
hannob 10y ago
> Anything related to privacy or security should be fully tested. How often do you write code that's not related to privacy or security? As soon as you connect something to the Internet it's related to privacy and security. The only situation where privacy and security don't matter a whole lot is if your code runs on airgapped devices with very limited tasks.
- EpicEng 10y ago
  I rarely write code that connects to anything, so almost never? There's a lot of code that's not written by web devs.
  - hannob 10y ago
    > I rarely write code that connects to anything, so almost never? Sounds hard to believe. What kind of code would that be? > There's a lot of code that's not written by web devs. There's also a lot more than the web that has some form of connectivity with the Internet (even if it's not directly connected it may still parse data that comes from untrusted sources). There is a widespread belief among many that "security is important, but doesn't matter for me". The most extreme example is obviously IoT ("Who would want to hack my coffeemaker?"), but there's a lot more. The unfortunate truth is: There is hardly any code these days that is not security relevant.

Touche 10y ago

The quote is not anti-test. You should read the article.

terryf 10y ago

The first part of your comment sounds really odd to me. I definitely write tests to check if my code works, among other things. Why do you think that is the wrong approach?

cheez 10y ago
I spent some time working for one of the biggest names in trading technology and one of the things I learned is that if your code doesn't work on more or less the first try, your design is probably wrong. The implications of that lesson are profound. When something doesn't work as expected, I now check my design and not the code.
- pc86 10y ago
  > if your code doesn't work on more or less the first try, your design is probably wrong Or you're tired. Or you're just not as focused as you could be. Or you have a deadline. Or you're trying something new. Or you're not fluent in the language yet. Or you're fluent in the language but not the framework. Or you're fluent in the language and framework but not the design pattern (if you use those). Or a whole host of other things. I'm not saying spot-checking the design isn't a good idea, but saying that it's the design more often than the code just doesn't match up with my experience.
  - cheez 10y ago
    The entire idea is to increase the probability that you are correct. So if you're coding tired, you're decreasing that probability. If you're using a framework you're not familiar with, you're decreasing that probability. This is why I mentioned that it was in trading technology: they only get paid to be right.
- lordnacho 10y ago
  How do you define "the first try"? Surely you're bound to have various trivial issues?
  - 10y ago

andrewstuart2 10y ago

And we write tests to exercise our own assumptions about how the system will be used. It's way too often I write some tests for my code, I quickly arrive at use cases and failure cases that I hadn't anticipated while writing for the optimal use case. Reality, on the other hand, is usually suboptimal.

Quarrelsome 10y ago

yes but if you follow that too strongly you end up having to rework tests _every_ time you change something which is absurd. You should be testing input/output and results as opposed to testing how the internal gubbins works. That's the line we have to carefully tread when making a test. The test shouldn't force the item to behave in directly the way it expects; more that the I/O is correct.

gr3yh47 10y ago
> yes but if you follow that too strongly you end up having to rework tests _every_ time you change something which is absurd. it depends how brittle your tests are. You can write tests that make sure internal stuff works at a unit level without them being so brittle to your point testing I/O or behavior is the way to accomplish this, but it can still be done at the level of internal functions/methods
demonshalo 10y ago
this is essentially the argument for functional tests over unit tests and while I generally agree, I think a mixture of the two is important. Unit tests should be used for extremely small and isolated mission critical objects while functional tests should generally cover the entirety of the I/O chain. That's how I do it at least and it works extremely well for a fraction of the cost!
- sethammons 10y ago
  How do you test error conditions? In my experience, you are typically not mocking hardly anything (ideally nothing) with functional tests and some error cases require mocking. I find unit tests helpful in this arena.
  - crdoconnor 10y ago
    I mock things all the time with functional tests. It makes it easier to reliably test your code's response to unusual conditions (e.g. error conditions) and it eliminates a source of brittleness in the test (you can write functional tests that hit the real paypal API and run them every day but every 2nd Friday they will fail because paypal is shit at keeping their servers up). This is in no way a benefit unique to unit tests.

raldi 10y ago

I agree with your assessment that the primary reason we write unit tests is to be able to quickly make changes in the future without fear of breaking something: You do some extra work today so that you can save a tiny bit of work (and worry) tomorrow, and over time that accumulated future benefit outweighs the penalty paid today. However, many in the industry forget that this is the underlying reasoning, as seen just two days ago here on this site. Read through the top-rated comments on this post: https://news.ycombinator.com/item?id=13119138 https://news.ycombinator.com/item?id=13119138 They describe an emergency situation where a single "3" needed to be changed to "4" ASAP or people would lose their jobs, and everyone's applauding the gatekeepers who insisted on significant refactoring and the creation of additional tests before the change could be approved. I agree with those who say those improvements should maybe have been demanded immediately after the fire was out, but those who would have delayed the firefighting out of blind allegiance to the rules seemed, to me, to have forgotten that the rules are there to serve the programmers (particularly, their ability to quickly ship working code), and not the other way around. A rule that's failing to do that should be changed or ignored.

wnevets 10y ago
Everything in moderation.
- codeduck 10y ago
  including moderation.
- nurettin 10y ago
  That's rather extreme.
- matthewmorgan 10y ago
  I prefer a slightly different translation of that Ancient Greek saying: nothing too much
- 0xdeadbeefbabe 10y ago
  For a moderate amount of everything.

gaius 10y ago

We write tests to be able to change it in the future with certain degree of confidence that we don't break anything and - if so - what exactly You don't want tests for that - you want a type system and a static analyzer.

cm2187 10y ago
But only tests will tell you you are calculating something correctly.
digi_owl 10y ago
But thats anathema to agile, yo. /s
raldi 10y ago
How does that confirm to you that, say, your ratelimiter code is correctly processing the first three requests in a given minute, rejecting the fourth, and approving a fifth once another minute has passed?
jon-wood 10y ago
A type system and static analysis certainly allow you to stop writing a certain category of test as you no longer need to assert that a method being passed an object of the wrong type copes with that. It doesn't completely relieve of you the need to test though as a type system won't assert that the code you're writing actually fulfils the businesses requirement that nothing can be ordered for same day shipping after 11am.
- gaius 10y ago
  You absolutely can do that with the type system, use a constrained subtype.
  - jon-wood 10y ago
    You've got my interest piqued now. How does that work in practice? And do you not still need to test that it complies with the actual business rules at some point?

fsloth 10y ago

"But we don't write tests to check if our code works." If the input data is complex enough to drive a non-trivial branching logic in our code - yes we do. Well, at least I do. The bonus is that it provides the fixed point for further changes at the same time.

misja111 10y ago

There are a couple of reasons why you should/ could write tests: - for catching regressions in the future - to verify that your code works. How else would you know that your backend service reacts properly on some edge condition - as documentation - as a kind of quality mark, for instance to be able to pass a code review or when writing code for an external party Unfortunately the last reason is also the most useless and still it is the one that seems to be the main motivation for many big enterprise developers.

TallGuyShort 10y ago
>> - as documentation Thank you for including this - it's underappreciated. So many times recently I've needed to write code against some poorly documented API (not always due to lack of effort, some things are hard to document well in prose / JavaDocs, or just due to constant change), but I took one look at the unit tests and it all made sense - and I knew it was up to date for the latest work.
- Freak_NL 10y ago
  Tests are great for dyslexic colleagues as well. A good clear concise unit test is often easier to grasp than the best of API documentation (e.g., JavaDoc in Java).
  - TallGuyShort 10y ago
    Good point - I'd bet it applies to autism and such too.
    - euyyn 10y ago
      I think it applies to everybody. I think it was Terence Tao who had a blog post claiming (and showing) that teaching by example first is the most effective.
mannykannot 10y ago
I know this is not formally an ordered list, but it looks cart-before-the-horse-ish to have catching future regressions before verifying that it works in the first place. (This comment is effectively a reply to the OC's claim that we don't test to verify.)

EGreg 10y ago

So then why don't you just write the tests BEFORE you are about to refactor something or change some interface? Why write them at "launch time" of the feature?

st3v3r 10y ago
Because if you haven't written them, odds are the code is not particularly well suited to be testable. Side effects out the wazoo, tight coupling, etc.

jsight 10y ago

I have seen codebases where the developers diligently insured that every getter and setter method had a test case. They had great test coverage, but not particularly reliable outputs. OTOH, projects with excellent end-to-end tests tend to have lower coverage but better regression management. I think this is the point of the article?

eveningcoffee 10y ago
I think that this is one of the points. I think that full test coverage is nor sufficient or necessary for the working software. What matters is the sufficient covering of the edge cases. I think that an additional point to keep in mind is that most probably Kent Beck talked mostly from the point of statically typed languages.

randomdata 10y ago

Technically, we write tests to document what the code does. We could use any language, including english, to do that. However, using a programming language carries the side benefit of being able to allow automatic verification that the documentation is at least in sync with the code (although not necessarily accurate with respect to the project goals).

sidlls 10y ago
And then who or what documents the tests? Test code is code. That's important and all too often that fact is sort of elided. "We" write comments and use "plain" language documentation, or formatted but otherwise plain language for documentation generators to parse to document our code, and that includes "what the code does." I'd argue that if your test code is what you're relying on to "document what the code does" then you are probably (almost certainly) over-testing, testing the wrong things, or some combination. Oh, and also using test code incorrectly (as documentation).

wvenable 10y ago

> We write tests to be able to change it in the future Which works great as long as your changes are shallow. If your changes aren't shallow then you have to change your tests as well and that defeats the purpose. Automated testing is good for freezing an interface and it's behavior; allowing you to change implementation while maintaining the same outputs. Lots of technology requires this: Networked API endpoints, libraries, etc. But most change I encounter is from changes in requirements that necessarily requires reworking and re-arranging code that won't be compatible with the test suite.

Jtsummers 10y ago
Different types of tests. You can have tests of the whole system which should always pass. Then there's unit tests, which may have to be removed/updated (though again, this depends on your scale of units). Some you expect to fail after a change to requirements get incorporated. Others you expect to pass, and you react accordingly.
olavk 10y ago
If the code is designed following the single responsibility principle, and tests are written to test behavior rather than implementation details, then each requirements change should only affect the tests which specifically test for the behavior which changes.
- wvenable 10y ago
  That's still a very narrow view of changes; the belief that you can isolate every part and that requirements changes would then only affect those isolated parts seems is pure fantasy. I've had to completely redesign entire subsystems; break down components into different pieces; move code between different layers; etc. Honestly as a manager nothing bothers me more than developers who try and patch complex changes into existing systems without re-thinking how it affects everything else. I have to re-factor a project that's a total mess because code was added but not removed or changed over the course of some very big requirements changes. I'm sure all the tests pass but it's impossible to follow now. The problem with unit tests is it adds an extra layer of friction on making changes that benefit the product. You are actively discouraged from changing your design from your initial assumptions! This change friction can be seen as a benefit if you need all your interfaces to be stable (like with a library). But it's a trade off and it's not appropriate everywhere.

gshulegaard 10y ago

This is the right answer. I will also point out that as a result of this property collaboration becomes easier.

vikiomega9 10y ago

N00b question, what's another technique? I mean if tests are assertions on assumptions is there something better?

0xdeadbeefbabe 10y ago

That only works if you run the tests often. Writing them is less important than running bits of your program often enough to discover how it really works.

mannykannot 10y ago

> But we don't write tests to check if our code works. So what methods do you use for that purpose?

partycoder 10y ago

Tests can be seen as a form of executable documentation, or a executable specification, a sort of self-enforcing contract.

hinkley 10y ago

When I start thinking of some of my code as safety equipment, and some like shop tools, a lot of my decision processes become more straightforward. My tests and practices are, occasionally, like a five point harness in a race car. Without it anchoring the driver in front of the wheel, they could never ever drive that fast with any safety at all. This one comes out as a counter to the 'straightjacket' argument some people like when the tools won't let them just write shit code that the rest of us have to babysit while they run off somewhere else to write more shit code. Which we are supposed to be grateful for... I digress. But most of the time my tests are more like smoke detectors. The ones that go off for no reason get replaced or removed. The ones that go off after I can see fire and smell smoke? Why am I paying for upkeep on those exactly? What's the value?

Steeeve 10y ago

Absolutely. The article itself (it's a short one) talks about doing just enough tests to have confidence in a product, and I think that's fantastic for early phase development. But when changes can come from anywhere (lots of developers potentially making changes) and the need for changes can be urgent, there's a great deal of value in complete code coverage. Not every developer thinks the same or has the same tendency for errors. I would caution people against thinking about tests as personal verification. With any luck, you'll be handing off that code base to someone else in time.

mrmrcoleman 10y ago

This is entirely subjective as it comes down to Kent's sense of what needs to be tested, which is arguably a better 'sense' than that of many other programmers. Do you allow your juniors to use their sense? No, you make them hit 100% coverage until they start to learn. As Paul McCartney said: "Learn the rules like a pro, so you can break them like an artist."

zebraflask 10y ago

That's not a bad perspective, but the counterpoint that I have seen several times is that the tests can tend to take on a split quality where some will insist that nothing can be changed if a test has to be changed along with it - and then the other camp steps in and doesn't want to put in the effort for new tests, so the existing ones get removed. Testing makes a lot of sense if you're doing data munging, but for front end or other kinds of code that are almost defined by their ability to create side effects, it's usually a waste of time. In my humble.