5 ms·

Can't agree more with this sentiment. Current software engineering practice has moved from focusing on the software itself to testing the software. Even peer re

by RyanZAG 14y ago

Can't agree more with this sentiment. Current software engineering practice has moved from focusing on the software itself to testing the software. Even peer reviews seem to be taking a back seat these days to 'make it pass the tests' engineering. 1) Your tests are just as likely to have bugs anyway. 2) The 'bad' bugs are logic bugs - if your logic is wrong, your tests are wrong, so you are testing the wrong thing. 3) Users often do things you didn't plan for. If you didn't plan for it, you didn't create a test for it. 100% code coverage does not mean you have tested every possible input, only that you've tested every possible branch. You can still go down the wrong branch/get the wrong result. 4) If you design first and test later, you're more likely to have designed real logic into your system. TDD will let you re-factor more easily - DDD (design driven development) will make it so you don't need to spend months re-factoring in the first place. Basically: software engineering in general needs more architecture and less ticking checkmarks - we are building space escalators here, we are not building chair legs. There is no need to re-invent the chair leg, "but this time with more tests".

nikita 14y ago
I definitely wouldn't argue with the fact that we have to think what we build, design carefully, etc. However imagine the world with massive test coverage that provides instant regression testing and the world without it. What would you choose? We do have thorough peer reviews, that's what Phabricator is for and nothing gets in without passing a code review. Some more comments. 1. True, but you have to hit a combination of a test bug/feature bug to let the feature bug slip through 2. Also true. And we had to deal with those type of bugs. When the area is tricky (transaction log rotation, quorums, etc) we sometime have to the extends of proving things formally. 3. Very true. And scenario based testing will only get you this far. Receiving customer feedback is critical for that. Obviously SAS has advantage here. 4. Everything is a tradeoff. We definitely design/prototype hard features before we build them I think the right way to put it "Once you put yourself on the scale between chair legs and a space ship, you should choose how you run the engineering process".
davidcuddeback 14y ago
I totally agree that tests aren't a replacement for thoughtful design, but neither is thoughtful design a replacement for tests. I think they complement each other. Thoughtful design allows you to model how you expect the software to work and your tests help you validate that model. 1) Your tests are just as likely to have bugs anyway. This is partially addressed by the "red-green" part of TDD's process. I catch many bugs in my unit tests by ensuring that the tests start out red and become green only after writing the code that's supposed to implement what's being tested. Yes, it's possible that some bugs will still exist in your test code, but I think it significantly reduces the probability that a bug exists in your application code. 4) If you design first and test later, you're more likely to have designed real logic into your system. TDD will let you re-factor more easily - DDD (design driven development) will make it so you don't need to spend months re-factoring in the first place. I agree with the first part, that putting some thought into design upfront is likely to improve the overall quality of your software, but I don't see how that necessarily implies "test last." I like Domain Driven Design [1] (another DDD) for upfront design, but I still use TDD when implementing the design. I've not heard of Design Driven Development before. Is that another name for Domain Driven Design? Its Wikipedia page [2] is lacking substance. A word about refactoring: I too find that putting thought into design before implementation reduces the need for large refactorings later. I agree with you there. However, I've never found that TDD helps me refactor "later." With TDD, I employ many micro-refactors during implementation. In other words, DDD provides the outline, TDD fills in the details. Alternatively: DDD is designing in-the-large, while TDD is designing in-the-small. [1] http://en.wikipedia.org/wiki/Domain-driven_design http://en.wikipedia.org/wiki/Domain-driven_design [2] https://en.wikipedia.org/wiki/Design-driven_development https://en.wikipedia.org/wiki/Design-driven_development
- RyanZAG 14y ago
  You and nikita below are both right that having tests is much better than not having tests, and the argument isn't really about choosing design or tests - you definitely want both. I was just trying to address the same point that HeyLaughingBoy above was addressing - tests are not a replacement for design; quality is not about tests. Tests are a tool to improve productivity in a large code base across many developers, and they're also a useful tool for a single programmer to use to focus their development. However, tests themselves are not a mark of quality, and any actual bugs that tests catch are just nice extras - and if you follow TDD, they will never catch bugs, since you will code until the test passes, bugs and all. So my point is very simple: quality is in design and implementation. Tests are just for increasing engineering efficiency - you can have the same quality without tests as long as you have enough time (manual QA). You can't have the same quality without design, even with all the tests in the world.
shurcooL 14y ago
Agree. IMO the main benefit/purpose of tests is to ensure you don't unintentionally break something later down the road, when making changes elsewhere. They're there to help you realize that something unexpected has changed. Secondary, they can help you verify the correctness of certain things that may go unnoticed (produce negative effects elsewhere) otherwise. But it's less useful to spend a lot time/effort on writing tests that verify things that are obvious to see anyway.
groby_b 14y ago
'make it pass the tests' engineering. I'm not aware of any credible engineering effort that only does that. 1) Your tests are just as likely to have bugs anyway. Yes, but only with P(test_bug). We see code bugs with P(code_bug), and if there are tests covering all those pieces of code, the final probability of bad things shipping to production is P(test_bug) * P(code_bug) Obviously, it's an overly optimistic assumption, but tests serve as one additional layer in a "defense in depth". 2) The 'bad' bugs are logic bugs - if your logic is wrong, your tests are wrong, so you are testing the wrong thing. If your logic is wrong, your architecture is likely to be wrong, too. If your basic assumptions are wrong, your code will be too, no matter what. 3) Users often do things you didn't plan for. Yes, and architecture isn't going to save you from that either. (It helps, though. Planning to minimize state is a good idea :) 4) [...] will make it so you don't need to spend months re-factoring in the first place. Which is a great strategy - if you can assume your requirements never change. If you work in a field like that, excellent. Most of us, however, need to deal with a rapidly changing field. software engineering in general needs more architecture Good software engineering has architecture. Tests augment it. How much architecture you create up front, how much you create on the fly, and how often it changes depends entirely on the field you work in, and how much resources you can spend on an issue.

by RyanZAG 14y ago

nikita 14y ago

I definitely wouldn't argue with the fact that we have to think what we build, design carefully, etc. However imagine the world with massive test coverage that provides instant regression testing and the world without it. What would you choose? We do have thorough peer reviews, that's what Phabricator is for and nothing gets in without passing a code review. Some more comments. 1. True, but you have to hit a combination of a test bug/feature bug to let the feature bug slip through 2. Also true. And we had to deal with those type of bugs. When the area is tricky (transaction log rotation, quorums, etc) we sometime have to the extends of proving things formally. 3. Very true. And scenario based testing will only get you this far. Receiving customer feedback is critical for that. Obviously SAS has advantage here. 4. Everything is a tradeoff. We definitely design/prototype hard features before we build them I think the right way to put it "Once you put yourself on the scale between chair legs and a space ship, you should choose how you run the engineering process".

davidcuddeback 14y ago

I totally agree that tests aren't a replacement for thoughtful design, but neither is thoughtful design a replacement for tests. I think they complement each other. Thoughtful design allows you to model how you expect the software to work and your tests help you validate that model. 1) Your tests are just as likely to have bugs anyway. This is partially addressed by the "red-green" part of TDD's process. I catch many bugs in my unit tests by ensuring that the tests start out red and become green only after writing the code that's supposed to implement what's being tested. Yes, it's possible that some bugs will still exist in your test code, but I think it significantly reduces the probability that a bug exists in your application code. 4) If you design first and test later, you're more likely to have designed real logic into your system. TDD will let you re-factor more easily - DDD (design driven development) will make it so you don't need to spend months re-factoring in the first place. I agree with the first part, that putting some thought into design upfront is likely to improve the overall quality of your software, but I don't see how that necessarily implies "test last." I like Domain Driven Design [1] (another DDD) for upfront design, but I still use TDD when implementing the design. I've not heard of Design Driven Development before. Is that another name for Domain Driven Design? Its Wikipedia page [2] is lacking substance. A word about refactoring: I too find that putting thought into design before implementation reduces the need for large refactorings later. I agree with you there. However, I've never found that TDD helps me refactor "later." With TDD, I employ many micro-refactors during implementation. In other words, DDD provides the outline, TDD fills in the details. Alternatively: DDD is designing in-the-large, while TDD is designing in-the-small. [1] http://en.wikipedia.org/wiki/Domain-driven_design http://en.wikipedia.org/wiki/Domain-driven_design [2] https://en.wikipedia.org/wiki/Design-driven_development https://en.wikipedia.org/wiki/Design-driven_development

RyanZAG 14y ago
You and nikita below are both right that having tests is much better than not having tests, and the argument isn't really about choosing design or tests - you definitely want both. I was just trying to address the same point that HeyLaughingBoy above was addressing - tests are not a replacement for design; quality is not about tests. Tests are a tool to improve productivity in a large code base across many developers, and they're also a useful tool for a single programmer to use to focus their development. However, tests themselves are not a mark of quality, and any actual bugs that tests catch are just nice extras - and if you follow TDD, they will never catch bugs, since you will code until the test passes, bugs and all. So my point is very simple: quality is in design and implementation. Tests are just for increasing engineering efficiency - you can have the same quality without tests as long as you have enough time (manual QA). You can't have the same quality without design, even with all the tests in the world.

shurcooL 14y ago

Agree. IMO the main benefit/purpose of tests is to ensure you don't unintentionally break something later down the road, when making changes elsewhere. They're there to help you realize that something unexpected has changed. Secondary, they can help you verify the correctness of certain things that may go unnoticed (produce negative effects elsewhere) otherwise. But it's less useful to spend a lot time/effort on writing tests that verify things that are obvious to see anyway.

groby_b 14y ago

'make it pass the tests' engineering. I'm not aware of any credible engineering effort that only does that. 1) Your tests are just as likely to have bugs anyway. Yes, but only with P(test_bug). We see code bugs with P(code_bug), and if there are tests covering all those pieces of code, the final probability of bad things shipping to production is P(test_bug) * P(code_bug) Obviously, it's an overly optimistic assumption, but tests serve as one additional layer in a "defense in depth". 2) The 'bad' bugs are logic bugs - if your logic is wrong, your tests are wrong, so you are testing the wrong thing. If your logic is wrong, your architecture is likely to be wrong, too. If your basic assumptions are wrong, your code will be too, no matter what. 3) Users often do things you didn't plan for. Yes, and architecture isn't going to save you from that either. (It helps, though. Planning to minimize state is a good idea :) 4) [...] will make it so you don't need to spend months re-factoring in the first place. Which is a great strategy - if you can assume your requirements never change. If you work in a field like that, excellent. Most of us, however, need to deal with a rapidly changing field. software engineering in general needs more architecture Good software engineering has architecture. Tests augment it. How much architecture you create up front, how much you create on the fly, and how often it changes depends entirely on the field you work in, and how much resources you can spend on an issue.