4 ms·
Testing is all well and good, but I worry that software is getting into the trap that Manufacturing learned to avoid decades ago: you can't test quality into t
by HeyLaughingBoy 14y ago
Testing is all well and good, but I worry that software is getting into the trap that Manufacturing learned to avoid decades ago: you can't test quality into the product.
Most blog posts about software quality seem to focus on testing. While testing is certainly a component of a good development process, it's really better to avoid injecting a defect than trying to be sure you execute tests to find it later.
- FourthProtocol 14y agoAgreed. If quality is critical (most OSs, for example, ship with known bugs) then look to embedded systems. You don't send the Mars Rover to Mars with known bugs in the software. Same for heart monitors. And that means waterfall, lots of modelling, simulation, and huge design up front.
- HeyLaughingBoy 14y agoIt definitely doesn't mean waterfall. Big Design Up Front is the way to go, but with caveats: only design as much as you need to, with the things you know are true or expect to happen. Requirements will change, no matter what, so you have to be flexible. Yes, it does mean simulation and modeling. Mainly to verify/disprove assumptions and learn how your system will work as early as possible. But waterfall? [shudder] No!!
- mahmoudimus 14y agoI think this is easier said than done. What I believe you're proposing is difficult to enforce. Can you give some examples -- sounds like you're mapping this to the manufacturing domain.
- HeyLaughingBoy 14y agoIn a nutshell, this is what we do. It's only difficult to enforce if the team doesn't buy into it. Dev takes a feature request off the list. Reads & understands the Requirement for the feature. If no Formal Requirement and the feature is complex enough to need one, then he meets with stakeholder and creates one. Meet with stakeholder & test person assigned again to review that the requirement as written is now correct, testable and edge cases are taken care of. Dev now designs the feature. Simple feature doesn't require any formal documentation, but we encourage at least a hallway discussion with someone so the design passes the smell test. Bigger/more important features need written design description and formal review of the design. Dev implements the feature, desk checks his code along the way for logic errors that the compiler or unit testing might miss and when satisfied, unit tests it. Code review is held. Possible discussion with tester about aspects the dev may have had difficulty testing, so the tester can focus on those areas. Tester may also need visibility into some hidden aspects (perhaps he wants to know the partial result of a calculation), so dev may have to modify code to log or print that somewhere so it can be verified. Tester creates test cases, reviews with developer to be sure they make sense and are testing things related to the feature, regression areas, etc. Tester runs test cases. If they all pass, feature is closed, test documentation is recorded. Yes, it seems like a lot, but all the reviews and checks mean that fewer bugs get to the point where a tester (or worse, a customer!) will find them. Testing becomes a way to verify that the software is working correctly instead of the way to find the bugs. Most bugs are due to misunderstanding a customer requirement so we have a focus on finding those misunderstandings or incomplete design problems before they turn into running code.
- RyanZAG 14y agoCan't agree more with this sentiment. Current software engineering practice has moved from focusing on the software itself to testing the software. Even peer reviews seem to be taking a back seat these days to 'make it pass the tests' engineering. 1) Your tests are just as likely to have bugs anyway. 2) The 'bad' bugs are logic bugs - if your logic is wrong, your tests are wrong, so you are testing the wrong thing. 3) Users often do things you didn't plan for. If you didn't plan for it, you didn't create a test for it. 100% code coverage does not mean you have tested every possible input, only that you've tested every possible branch. You can still go down the wrong branch/get the wrong result. 4) If you design first and test later, you're more likely to have designed real logic into your system. TDD will let you re-factor more easily - DDD (design driven development) will make it so you don't need to spend months re-factoring in the first place. Basically: software engineering in general needs more architecture and less ticking checkmarks - we are building space escalators here, we are not building chair legs. There is no need to re-invent the chair leg, "but this time with more tests".
- nikita 14y agoI definitely wouldn't argue with the fact that we have to think what we build, design carefully, etc. However imagine the world with massive test coverage that provides instant regression testing and the world without it. What would you choose? We do have thorough peer reviews, that's what Phabricator is for and nothing gets in without passing a code review. Some more comments. 1. True, but you have to hit a combination of a test bug/feature bug to let the feature bug slip through 2. Also true. And we had to deal with those type of bugs. When the area is tricky (transaction log rotation, quorums, etc) we sometime have to the extends of proving things formally. 3. Very true. And scenario based testing will only get you this far. Receiving customer feedback is critical for that. Obviously SAS has advantage here. 4. Everything is a tradeoff. We definitely design/prototype hard features before we build them I think the right way to put it "Once you put yourself on the scale between chair legs and a space ship, you should choose how you run the engineering process".
- davidcuddeback 14y agoI totally agree that tests aren't a replacement for thoughtful design, but neither is thoughtful design a replacement for tests. I think they complement each other. Thoughtful design allows you to model how you expect the software to work and your tests help you validate that model. 1) Your tests are just as likely to have bugs anyway. This is partially addressed by the "red-green" part of TDD's process. I catch many bugs in my unit tests by ensuring that the tests start out red and become green only after writing the code that's supposed to implement what's being tested. Yes, it's possible that some bugs will still exist in your test code, but I think it significantly reduces the probability that a bug exists in your application code. 4) If you design first and test later, you're more likely to have designed real logic into your system. TDD will let you re-factor more easily - DDD (design driven development) will make it so you don't need to spend months re-factoring in the first place. I agree with the first part, that putting some thought into design upfront is likely to improve the overall quality of your software, but I don't see how that necessarily implies "test last." I like Domain Driven Design [1] (another DDD) for upfront design, but I still use TDD when implementing the design. I've not heard of Design Driven Development before. Is that another name for Domain Driven Design? Its Wikipedia page [2] is lacking substance. A word about refactoring: I too find that putting thought into design before implementation reduces the need for large refactorings later. I agree with you there. However, I've never found that TDD helps me refactor "later." With TDD, I employ many micro-refactors during implementation. In other words, DDD provides the outline, TDD fills in the details. Alternatively: DDD is designing in-the-large, while TDD is designing in-the-small. [1] http://en.wikipedia.org/wiki/Domain-driven_design http://en.wikipedia.org/wiki/Domain-driven_design [2] https://en.wikipedia.org/wiki/Design-driven_development https://en.wikipedia.org/wiki/Design-driven_development
- alexrson 14y agoTesting is just one layer or defense against poor quality.