5 ms·

Practical Techniques to Achieve Quality in Large Software Projects

76 points by nikita 14y ago

smm2000 14y ago
I am curious how many unittests they have - hundreds or thousands...
- deleted 14y ago
  [deleted]
- nikita 14y ago
  We have thousands of C++ unit tests and hundreds of thousands of functional SQL tests
known 14y ago
Agile?
- nikita 14y ago
  We are not following any specific methodology such as agile. But we definitely pay a lot of attention to the varios engineering ideas and practices.
mahmoudimus 14y ago
Interesting, thanks for sharing! Can you share more about psyduck? How much time was invested in building out your testing systems?
- nikita 14y ago
  I would say it was an initial push on 1.5 months of one engineer with 10% of time of one engineer for maintenance. As the load increases we have to invest a few days here and there to scale the system.
HeyLaughingBoy 14y ago
Testing is all well and good, but I worry that software is getting into the trap that Manufacturing learned to avoid decades ago: you can't test quality into the product. Most blog posts about software quality seem to focus on testing. While testing is certainly a component of a good development process, it's really better to avoid injecting a defect than trying to be sure you execute tests to find it later.
- FourthProtocol 14y ago
  Agreed. If quality is critical (most OSs, for example, ship with known bugs) then look to embedded systems. You don't send the Mars Rover to Mars with known bugs in the software. Same for heart monitors. And that means waterfall, lots of modelling, simulation, and huge design up front.
  - HeyLaughingBoy 14y ago
    It definitely doesn't mean waterfall. Big Design Up Front is the way to go, but with caveats: only design as much as you need to, with the things you know are true or expect to happen. Requirements will change, no matter what, so you have to be flexible. Yes, it does mean simulation and modeling. Mainly to verify/disprove assumptions and learn how your system will work as early as possible. But waterfall? [shudder] No!!
- mahmoudimus 14y ago
  I think this is easier said than done. What I believe you're proposing is difficult to enforce. Can you give some examples -- sounds like you're mapping this to the manufacturing domain.
  - HeyLaughingBoy 14y ago
    In a nutshell, this is what we do. It's only difficult to enforce if the team doesn't buy into it. Dev takes a feature request off the list. Reads & understands the Requirement for the feature. If no Formal Requirement and the feature is complex enough to need one, then he meets with stakeholder and creates one. Meet with stakeholder & test person assigned again to review that the requirement as written is now correct, testable and edge cases are taken care of. Dev now designs the feature. Simple feature doesn't require any formal documentation, but we encourage at least a hallway discussion with someone so the design passes the smell test. Bigger/more important features need written design description and formal review of the design. Dev implements the feature, desk checks his code along the way for logic errors that the compiler or unit testing might miss and when satisfied, unit tests it. Code review is held. Possible discussion with tester about aspects the dev may have had difficulty testing, so the tester can focus on those areas. Tester may also need visibility into some hidden aspects (perhaps he wants to know the partial result of a calculation), so dev may have to modify code to log or print that somewhere so it can be verified. Tester creates test cases, reviews with developer to be sure they make sense and are testing things related to the feature, regression areas, etc. Tester runs test cases. If they all pass, feature is closed, test documentation is recorded. Yes, it seems like a lot, but all the reviews and checks mean that fewer bugs get to the point where a tester (or worse, a customer!) will find them. Testing becomes a way to verify that the software is working correctly instead of the way to find the bugs. Most bugs are due to misunderstanding a customer requirement so we have a focus on finding those misunderstandings or incomplete design problems before they turn into running code.
Ixiaus 14y ago
I absolutely hate it when I'm reading an interesting article on a startup/company blog and I go to click on their logo/name in the top left corner and it...takes me back to their blog instead of their homepage!
deleted 14y ago
[deleted]
- _pius 14y ago
  I have a friend who can derive no value from unit tests. He works on a thing that--by no fault of his own, but just by the very nature of physical reality--cannot be automatically tested. What is he, a quantum cryptographer?
urbina1985 14y ago
Test (and Design) are part of the development strategy. Depending on what your business goals are, you should optimize Design\Development\Test for it. Sometimes these goals are principals that you want to brand around (e.g. robustness for a DB server product) or tactical decision that adapt to a stage (catching up, learning, leapfrogging, supportability, locking a market). These are scenarios that I've seen (and some of them worked on) over the last years: * Deliver a strong v1 too late * Deliver a small v1 that outs a new market and allows competirors to catch up * Develop a vnext backcompat-focused product and be surpassed by a new\exisiting competitor * Develop a heavily customer focus product and get trapped in impossible\contradicting requirements * Deliver a weak v1 and lose market credibility * Deliver a secret v1 and have no real market for it If you have a clear view of the business goals, Test becomes less an opinion and more of a resourcing puzzle (still hard though). It's a great world of tradeoffs between layering tests, testability features, customer research\partnership\simulation, fast development, legacy ownership, responsibility distribution, exhaustiveness, documentation, community\customer engagement, reusability of tests, reusability of product code.

76 points by nikita 14y ago

smm2000 14y ago

I am curious how many unittests they have - hundreds or thousands...

deleted 14y ago
[deleted]
nikita 14y ago
We have thousands of C++ unit tests and hundreds of thousands of functional SQL tests

known 14y ago

Agile?

nikita 14y ago
We are not following any specific methodology such as agile. But we definitely pay a lot of attention to the varios engineering ideas and practices.

mahmoudimus 14y ago

Interesting, thanks for sharing! Can you share more about psyduck? How much time was invested in building out your testing systems?

nikita 14y ago
I would say it was an initial push on 1.5 months of one engineer with 10% of time of one engineer for maintenance. As the load increases we have to invest a few days here and there to scale the system.

HeyLaughingBoy 14y ago

Testing is all well and good, but I worry that software is getting into the trap that Manufacturing learned to avoid decades ago: you can't test quality into the product. Most blog posts about software quality seem to focus on testing. While testing is certainly a component of a good development process, it's really better to avoid injecting a defect than trying to be sure you execute tests to find it later.

FourthProtocol 14y ago
Agreed. If quality is critical (most OSs, for example, ship with known bugs) then look to embedded systems. You don't send the Mars Rover to Mars with known bugs in the software. Same for heart monitors. And that means waterfall, lots of modelling, simulation, and huge design up front.
- HeyLaughingBoy 14y ago
  It definitely doesn't mean waterfall. Big Design Up Front is the way to go, but with caveats: only design as much as you need to, with the things you know are true or expect to happen. Requirements will change, no matter what, so you have to be flexible. Yes, it does mean simulation and modeling. Mainly to verify/disprove assumptions and learn how your system will work as early as possible. But waterfall? [shudder] No!!
mahmoudimus 14y ago
I think this is easier said than done. What I believe you're proposing is difficult to enforce. Can you give some examples -- sounds like you're mapping this to the manufacturing domain.
- HeyLaughingBoy 14y ago
  In a nutshell, this is what we do. It's only difficult to enforce if the team doesn't buy into it. Dev takes a feature request off the list. Reads & understands the Requirement for the feature. If no Formal Requirement and the feature is complex enough to need one, then he meets with stakeholder and creates one. Meet with stakeholder & test person assigned again to review that the requirement as written is now correct, testable and edge cases are taken care of. Dev now designs the feature. Simple feature doesn't require any formal documentation, but we encourage at least a hallway discussion with someone so the design passes the smell test. Bigger/more important features need written design description and formal review of the design. Dev implements the feature, desk checks his code along the way for logic errors that the compiler or unit testing might miss and when satisfied, unit tests it. Code review is held. Possible discussion with tester about aspects the dev may have had difficulty testing, so the tester can focus on those areas. Tester may also need visibility into some hidden aspects (perhaps he wants to know the partial result of a calculation), so dev may have to modify code to log or print that somewhere so it can be verified. Tester creates test cases, reviews with developer to be sure they make sense and are testing things related to the feature, regression areas, etc. Tester runs test cases. If they all pass, feature is closed, test documentation is recorded. Yes, it seems like a lot, but all the reviews and checks mean that fewer bugs get to the point where a tester (or worse, a customer!) will find them. Testing becomes a way to verify that the software is working correctly instead of the way to find the bugs. Most bugs are due to misunderstanding a customer requirement so we have a focus on finding those misunderstandings or incomplete design problems before they turn into running code.

Ixiaus 14y ago

I absolutely hate it when I'm reading an interesting article on a startup/company blog and I go to click on their logo/name in the top left corner and it...takes me back to their blog instead of their homepage!

deleted 14y ago

[deleted]

_pius 14y ago
I have a friend who can derive no value from unit tests. He works on a thing that--by no fault of his own, but just by the very nature of physical reality--cannot be automatically tested. What is he, a quantum cryptographer?

urbina1985 14y ago

Test (and Design) are part of the development strategy. Depending on what your business goals are, you should optimize Design\Development\Test for it. Sometimes these goals are principals that you want to brand around (e.g. robustness for a DB server product) or tactical decision that adapt to a stage (catching up, learning, leapfrogging, supportability, locking a market). These are scenarios that I've seen (and some of them worked on) over the last years: * Deliver a strong v1 too late * Deliver a small v1 that outs a new market and allows competirors to catch up * Develop a vnext backcompat-focused product and be surpassed by a new\exisiting competitor * Develop a heavily customer focus product and get trapped in impossible\contradicting requirements * Deliver a weak v1 and lose market credibility * Deliver a secret v1 and have no real market for it If you have a clear view of the business goals, Test becomes less an opinion and more of a resourcing puzzle (still hard though). It's a great world of tradeoffs between layering tests, testability features, customer research\partnership\simulation, fast development, legacy ownership, responsibility distribution, exhaustiveness, documentation, community\customer engagement, reusability of tests, reusability of product code.