5 ms·
I'll put the question to the other readers: How often do you find, despite having written tests, that there is some bug in your software? And how many of those
by lordnacho 10y ago
I'll put the question to the other readers:
How often do you find, despite having written tests, that there is some bug in your software? And how many of those times did you think that you should have considered it beforehand, rather than that it would be impossible to foresee?
In my experience the most useful tests are the ones that came from some unforeseen bug, which was then fixed and a test case built around it, so that it wouldn't get "unfixed".
The least useful tests are the ones for cases you know not to invoke, because they are obvious. Like how you know when you divide by a variable, you know it can't be zero. So you make sure it can't be zero, making the test case a bit moot.
- emodendroket 10y agoThere should be a law banning people from asserting the constants have the expected value in tests.
- andy_ppp 10y agoTrue ;-)
- wjnc 10y agoDo you mean test explicitly and fail graciously? So even test over false inputs, but check that the code fails with the right exit?
- emodendroket 10y agoI mean if you have a method whose definition is literally just "return 'abc'" and then you have a test assert that that method returns 'abc', knock it off.
- jdmichal 10y agoI call those types of test, "testing that the compiler works." Not really something you should be testing! It also applies to testing code that is just a wrapper around third-party libraries.
- remar 10y agoI was taught that there should be tests against things like class constants so that you have a failing build in the event that someone changes a const that's part of the specification. This was from some TDD course that my employer brought in where they emphasized that consts that are part of a specification or requirement should be tested against to prevent them from being accidentally changed or changed without full consideration of side effects/review against the program spec. Is this what you're referring to?
- emodendroket 10y agoI guess so but I disagree with that teaching. All it does is make me change it in two places.
- nostrademons 10y agoThat's sorta the point - to make you think before you change. A constant without a test is part of the implementation, and you can refactor as necessary to make the implementation perform better. A constant with a test is part of the interface, and if you change it, client behavior has changed and the tests should change with it.
- emodendroket 10y agoYes, OK, but what I'm telling you is I think it's just as mindless. If anything, I look less at the tests when I know I'm going to break fifty of them with any real change at all. "Yeah, yeah, whatever" and copying and pasting the new values is the more likely behavior in this scenario.
- aindhaden 10y agoThis is true only for a very particular kind of constants, like the ones defining API error codes (eg. define PAGE_NOT_FOUND 404). And those should be covered by integration tests, not unit tests.
- dosethree 10y agoI TDD a fair bit, and both kinds of tests are useful. I do really love the tests that I write to replicate a regression bug. The other types of tests are more like living documentation of my internal APIs for future developers, and are still useful.
- humanrebar 10y agoYes, and a special case of this, which is rarely correct in "I don't get paid for tests" code, is tests that make sure errors are reported or handled correctly. It's rarely a good idea to ship a NullPointerException just to see if you correctly turn it into an error code or something.
- andrewstuart2 10y agoPerhaps I'm just an outlier, but I think every time I've gone in to write tests for my code (usually post-testing rather than red/green testing), it's rather quickly invalidated some assumptions I made when writing the tests. "Oh yeah, that's going to need to be thread safe." "Oh yeah, that might be nil. Rather frequently." In my experience, testing has been much more beneficial as an exercise of my mental model of the code and its interaction than for refactoring. But to that end, I can think of quite a few times that I've been very grateful for a unit test suite while I did a large refactoring.
- Klathmon 10y agoI've had a very similar experience. It's worth it's weight in gold when you can flesh out edge cases and logic issues before going all-in on an architecture, and being able to refactor without fear is a great bonus, but it's not always applicable. Unless the majority of the tests are "external" (i.e. testing an API by calling it's endpoints and testing the return), a major refactor is going to require updated tests, and it's all too common that I see people take the "easy" way out and fix the tests to conform to the way it's working after the refactor rather than the way it should work.
- bluesign 10y agoI agree, tread safety and race conditions I got hit a lot, other than that test coverage provided a lot I guess
- Sean1708 10y ago> tread safety and race conditions I feel like this comment wouldn't be out of place in an F1 thread.
- hoorayimhelping 10y ago>Perhaps I'm just an outlier If you are, then I am as well, because this is where a huge part of the value of tests come from - validating assumptions early and quickly and cheaply.
- wolfgang42 10y agoI've found that this depends heavily on the software in question. For most of what I do (basic line-of-business type webapps & servers) the vast majority of the code works on the first try and is obvious enough that I never break it; during development there tends to be a few sections that break repeatedly while I'm changing other things, and those are the sections I write tests for. (Generally this is a few percent of the entire codebase.) On the other hand, more 'computer-sciencey' projects tend to need a correspondingly larger proportion of test cases. The most dramatic example of this was when I built a Lisp interpreter as a side project: this was the only software I've ever written fully test-first, and I don't think I would have been able to get it working at all without a full test suite that hit every line of code at least twice.
- marcosdumay 10y agoAs a follow-up question, did you (or could you have) develop that lisp interpreter on the purest TDD approach of: - Write a few tests that fail - Write code until your tests stop failing - Repeat until the program is done? In my experience, problems that require writing the tests first normally require writing all the tests first. If you start solving only a subset of them, you will have to rewrite most of the code once you start looking for the other tests.
- wolfgang42 10y ago> did you (or could you have) develop that lisp interpreter on the purest TDD approach Yes, that's what I meant by "fully test-first". I was following along with Paul Graham's The Roots of Lisp[1], so I had a convenient set of pre-made test cases. I would start by copying his examples into a test case: it('exec atom on a symbol is true', () => { var symbol = Symbols.get_symbol_named('foo') expect(exec(empty_scope, [natives.atom, [natives.quote, symbol]])).to.equal(true) }) it('exec atom on a non-empty list is false', () => { expect(exec(empty_scope, [natives.atom, [natives.quote, ['a']]])).to.equal(false) }) it('exec atom on an empty list is true', () => { expect(exec(empty_scope, [natives.atom, [natives.quote, []]])).to.equal(true) }) And then write my code: natives.atom = new Native(function (item) { var i = exec(this, item) return (i instanceof Array && i.length == 0) || Symbols.is_symbol(i) }) Once I got through the paper that way, I had a complete Lisp interpreter working, and wrote a few small programs in it with no trouble at all. > In my experience, problems that require writing the tests first normally require writing all the tests first. If you start solving only a subset of them, you will have to rewrite most of the code once you start looking for the other tests. This has been my experience as well. As I say, this is the only program I've ever written fully test-first; I think it was only possible because I already knew exactly what the inputs and outputs were going to be (having them well-specified by virtue of being a Lisp) and which ones I needed to implement to be able to get later ones working (with the help of the paper). Generally I find it's not possible to work this way; my usual approach for solving problems where the code isn't immediately obvious is to write (and generally rewrite) in parallel: - A set of usage examples and/or documentation, to clarify what exactly I'm trying to do and make sure the interface makes sense. - The actual implementation (or, in early stages, some pseudocode). - The tests (if any), to check that what I've written actually works. Often these are the same as the usage examples. Frequently questions raised while writing one of these will influcence the others, often significantly; often I'll only think of an edge case while I'm writing the code and then have to go back and add it to the tests, or realize that the code could be simplified by changing the way the API works. [1]: http://www.paulgraham.com/rootsoflisp.html http://www.paulgraham.com/rootsoflisp.html
- lmm 10y agoI see bugs in my code very rarely nowadays, and they're always things I should have seen beforehand. In my experience if you're seeing edge cases at all they're a sign that your model is wrong - the business processes we're replacing with software are always simple, there's really no reason for code to ever be complex. The problems we get are those we introduce for ourselves by overcomplicating things in code. I do think of myself as using TDD, but I use the tests to drive the model and often end up not needing the test - a bit like https://spin.atomicobject.com/2014/12/09/typed-language-tdd-part1/ https://spin.atomicobject.com/2014/12/09/typed-language-tdd-... . With experience, more and more I'll skip past the writing a test and deleting it after, and go straight to encoding the properties that I actually need in my types.
- peeters 10y agoTesting for me is never primarily about getting it right the first time. Although writing tests in parallel with coverage can definitely help to validate your assumptions, particularly with negative testing that you might not test manually. The greater value of the tests is later on, when you need to make a change, and want to know whether you broke some existing functionality. Or, once you have a framework of tests around something, being able to quickly write a test that fails due to a discovered bug, and knowing that you fixed it by passing the test.
- enraged_camel 10y ago>>The least useful tests are the ones for cases you know not to invoke, because they are obvious. Maybe you know not to invoke them because you wrote those features, but what about the rest of the team? Or the poor schmuck who takes over your codebase after you leave? Test cases help future-proof your code against the unknowns and the unexpected. I think of them as guard rails on a path. When the path is on straight terrain, they aren't necessary. When it is crossing the side of a cliff though, you're glad they're there.
- bpchaps 10y agoFrom the receiving end (long history of "ops" work), I've seen that a lot of the code itself tends to come out really well and the developers who write the code are a lot happier. That said, many of the feature requests from the ops' side end up taking an extremely long time to produce, if ever. Very basic requests that would save the team dozens of hours per week go on and on longer than they should. In these sorts of shops, the ops side ends up pretty miserable and builds up a lot of frustration for the development team, which drives the two groups apart. More recently, my team's been asked the ops team is being asked to write tests for ops' prod bugfix automation. Categorically speaking, we're not developers, so it tends to take us longer to write tests than the devs who write them daily. Issues pile up and span across months, rather than days. The reasoning behind the tests is understandable since it helps build assurance that our scripts won't break the world. Though, the ops team is battle hardened enough to know that if we break it, we have to fix it - it's in our best interest to write code that's reliable as possible. I don't think any of us would be in our roles if we didn't understand it. Frustrating stuff.
- bluesign 10y agoSimilar question can be, which kind of tests failing most frequently, this is a good insight on what kind of tests you should focus on your team
- metaphorm 10y agoyou're describing two different types of tests. one kind is the acceptance test, where you write a test that specifies correct behavior under normal circumstances (including expected error scenarios). the other kind is a regression test, where you reproduce a discovered bug in a test environment, then fix it, and make sure that the test covers the fixed code to confirm that the bug is no longer occurring. both types of tests are important and for different reasons. they aren't mutually exclusive and they aren't addressing the same problem.