10 ms·

Ask HN: How did NASA make reliable software if they didn't invent unit tests?

98 points by sebastianconcpt 10y ago

PaulHoule 10y ago
Code review.
nekopa 10y ago
Process too.
hollander 10y ago
Back in the times when we counted in bytes or kilobytes, programs were simpler to manage. One person could overview all code. Now we have programs with million lines. Nobody can oversee that.
- aardvark291 10y ago
  "One person could overview all code." Not sure that's true. Imagine reviewing the code of the Apollo Guidance Computer, just to take an example from the 60's. The stack of paper containing the source code printout is more than five feet tall: http://skeptics.stackexchange.com/questions/31602/is-this-a-photo-of-margaret-hamilton-standing-next-to-apollo-project-code-that-s http://skeptics.stackexchange.com/questions/31602/is-this-a-...
  - nostrademons 10y ago
    It's on GitHub: https://github.com/chrislgarry/Apollo-11 https://github.com/chrislgarry/Apollo-11 Reviewing it would be non-trivial, but in terms of absolute code size, it's not much larger than a moderately-popular open-source project today. It's significantly smaller than React, for example: https://github.com/facebook/react/tree/master/src https://github.com/facebook/react/tree/master/src Edit: Downloaded both repositories. 'wc' puts the AGC at roughly 60K LOC, while just the 'src' directory in React is 70K.
yeukhon 10y ago
My wild guess is just a lot of testing. Rigorous testing. Note a lot of early NASA code were assembly.
typetypetype 10y ago
I think you mean to say "automated, script-based unit tests". You can test code manually.
- jpindar 10y ago
  Back in the day, people used to do something called "desk checking".
efnx 10y ago
A while back someone posted some programming guidelines issued by nasa. It detailed requiring assert statements at least every 10 loc. It spoke of many of other things as well, but that was one that stuck in my head.
- athenot 10y ago
  For whatever it's worth, assert statements were also heavily used in embedded digital TV software. The philosophy is that things either work or they don't. With enough QA and field testing, those asserts will show up if state goes awry.
- pkaye 10y ago
  This is something I do myself. As I build up the code, I write tons of self checking asserts so it fails quite quickly on the first sign of something wrong. At some point, I'll remove the less likely asserts as they just clutter up the code.
- jdavis703 10y ago
  Is this it this: http://pixelscommander.com/wp-content/uploads/2014/12/P10.pdf http://pixelscommander.com/wp-content/uploads/2014/12/P10.pd...? If so it looks like functions can't be more than 60 lines (rule #4), and that the "assertion density" (rule #5) must be two asserts per function.
deleted 10y ago
[deleted]
geonnave 10y ago
Some thoughts I have: - I think they did apply methodologies of reliability that worked in hardware, to software (e.g redundancy, automatic fault detection/tracing/correction) - Also, in the early days, software was so tied to hardware, and hardware was faaar less complex, so they probably could understand everything that was happening in the system at a given time
noblethrasher 10y ago
This article by Walter Bright (creator of D, also hangs out on HN as 'WalterBright) might be illuminating as well as instructive: http://www.drdobbs.com/architecture-and-design/safe-systems-from-unreliable-parts/228701716 http://www.drdobbs.com/architecture-and-design/safe-systems-...
wwkeyboard 10y ago
In addition to everything else mentioned here they spent a TON of money[1]. For example they spent what would be 3.6 billion 2016 dollars on Guidance and Navigation. If your current project could drop a few hundred million on QC I bet quality would go way up regardless of automated testing or not. [1]http://history.nasa.gov/SP-4029/Apollo_18-16_Apollo_Program_Budget_Appropriations.htm http://history.nasa.gov/SP-4029/Apollo_18-16_Apollo_Program_...
- kakali 10y ago
  I think your number includes design and building one of the first solid state computers.
  - wwkeyboard 10y ago
    The "few hundred million for quality control" number still stands. In addition they were able to draw on everyone from the people designing the transistors through to the programers to debug problems. Not something many modern projects can do.
abtinf 10y ago
Unit testing is not a method of achieving reliability, but of accelerating development. They do not prove that a system functions correctly, which is more in the domain of systems and integration testing. Also, testing is only one small part of building reliable systems. Instead, reliability requires actual engineering practice, which is non-existent in the software world except in a few life-critical domains. NASA and their contractor's software engineering practices have been covered extensively elsewhere. Here are some links. https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff http://history.nasa.gov/computers/Ch4-5.html http://history.nasa.gov/computers/Ch4-5.html
- fokov 10y ago
  Unit tests do not even prove that unit functions correctly either. Code coverage doesn't necessarily mean anything too.
- rootusrootus 10y ago
  My takeaway from reading the fastcompany.com article is that it's not process improvement per se that provides quality software, it's money. NASA has invested a great deal more time, and therefore money, into developing software than a typical commercial software company can afford. We will need quite a lot more programmers necessarily getting paid a lot less if we want commercially viable software production within spitting distance of NASA quality expectations.
  - Spearchucker 10y ago
    That's just it. If a bug in software can cause death you're not going to use agile. You're going to use a waterfall-like process, and you're going to have stringent quality gates at every single transition - from envisioning, through requirements analysis, design, development, test, release, and into operation. It's the only way. I know because I built a military guaranteed messaging system. Tech aspect was small, but it took 24 months.
    - e12e 10y ago
ThomPete 10y ago
I think it's important to remember that the kind of software written for any kind of flight control or guidance system are very very very far from the kind of dynamic software normally done today. There where more likely to be hardware errors than software errors, thats how close to the hardware they wrote.
invaliduser 10y ago
Unit Tests are not the one and only metrics for software quality. Believe it or not, before unit tests were coined, we managed to build reliable software too. Actually a lot of the internet was actually built that way. But your assumption is very interesting, because it states a very common misconception of quality (whatever that means, reliability, maintainability, resilience, etc): * "Unit tests imply reliable software". Well, it does not, you'd need reliable unit tests for that, and from what I've seen in the industry, not every unit test out in the wild increases reliability. * "Without unit tests, you can't have quality software". That is just not true either. First, not all tests are unit tests, end-to-end (integration/functional/whatever) test are extremely useful and raises the quality too. Even without any automated test, the quality of documentation, contract programming, manual testing, etc also help a lot. Also, note that apart from testing, a lot of different practices contribute to reliability. Separation of concerns, the language level of abstraction, the overall experience and involvements of the developers, and many others.
- jeremiep 10y ago
  I find that the people who are the most obsessed about unit testing generally aren't good programmers to begin with. To me, keeping code simple and easy to reason about is what yields reliable software. If you can't wrap your head around something in the first place then tests are only going to give you the illusion of reliability. Its as Rich Hickey put it: "What does every bug have in common? They all passed the type checker and the tests!"
  - Cacti 10y ago
    As the saying goes, crap in, crap out. It's just as easy to write crappy unit tests as it is crappy code.
    - tmm 10y ago
      Or, as W. E. Deming wrote: "You cannot inspect quality into a product."
    - finishingmove 10y ago
tuned 10y ago
Asserting about involved variables each 10 lines of code (:
carapace 10y ago
Margaret Hamilton
- nickpsecurity 10y ago
  I suggest always including links when you drop her name as significance will be lost on many. https://en.wikipedia.org/wiki/Margaret_Hamilton_(scientist) https://en.wikipedia.org/wiki/Margaret_Hamilton_(scientist) http://htius.com/ http://htius.com/
  - carapace 10y ago
    I agree, I was trying to be "arch" ;-) Here's a good synopsis: http://www.history.nasa.gov/alsj/a11/a11Hamilton.html http://www.history.nasa.gov/alsj/a11/a11Hamilton.html """ NASA Press Release, 3 September 2003 NASA HONORS APOLLO ENGINEER Margaret Hamilton, leader of the team that developed the flight software for the agency's Apollo missions, has been granted a NASA Exceptional Space Act Award for her scientific and technical contributions. "The Apollo flight software Ms. Hamilton and her team developed was truly a pioneering effort," said NASA Administrator Sean O'Keefe. "The concepts she and her team created became the building blocks for modern 'software engineering.' It's an honor to recognize Ms. Hamilton for her extraordinary contributions to NASA," he said. Dr. Paul Curto, senior technologist for NASA's Inventions and Contributions Board nominated Hamilton for the award. Curto said, "I was surprised to discover she was never formally recognized for her groundbreaking work. Her concepts of asynchronous software, priority scheduling, end-to-end testing, and man-in-the-loop decision capability, such as priority displays, became the foundation for ultra-reliable software design." One example of the value of Hamilton's software work occurred during the Apollo 11 mission. Approximately three minutes before Eagle's touchdown on the moon, the software over rode a command to switch the flight computer's priority processing to a radar system whose 'on' switch had been manually activated due to a faulty written operations script provided to the crew. The action by the software permitted the mission to safely continue. """
  - sebastianconcpt 10y ago
    Indeed. Thank you for the links
dsl 10y ago
Your submission falsely implies that unit tests create reliable software. But that aside... Avionics software is subjected to multiple rounds of peer review (often by different companies), integration testing, and in some cases multiple different implementations are developed and black box tested against each other for different outputs given the same inputs.
owiejg 10y ago
I think they spend a lot of time doing very detailed code reviews.
njharman 10y ago
Unit tests aren't really about testing (in the validating sense) software. They in fact are quite deficient in that role. No integration tests, no end-to-end tests, no fuzz testing, etc. Unit Tests are foremost a software development tool. They force (in order to be unit testable) separation of concerns, definition of interfaces (i.e. the "contract" with function's users), etc. Mechanically, they also enable validating that internal refactoring has not violated that contract.
- gwbas1c 10y ago
  As big of a fan as I am about unit tests, I find their value somewhat dubious. They take a long time to write, and when they fail, it often means the unit test needs to be updated. Mocking interactions among objects takes a long time. What I find has a better ROI is using the same unit testing tools, but writing what unit test purists will call integration tests. Basically, using unit test tools where most of a program is instantiated, but I/O is mocked, is much more useful. When these tests fail, it usually indicates a bug instead of a part of a unit test that needs to be updated.
  - flukus 10y ago
    It sounds like a problem with your tests actually. Unit test shouldn't take a long time to write and the should only fail for a very specific.
  - njharman 10y ago
    This is why/how they are a development tool. "Mocking interactions among objects", often breaking, taking too long to write (I expect to spend 30% dev on units, 30% on code rest thinking, planning, etc). Those are all indicators that your code/architecture is bad. Too complex, too coupled, too much concern, not layered into clean interfaces, etc.
daxfohl 10y ago
Who invented unit tests? Does that person have a monopoly on highly reliable software? What an awesome patent trolling opportunity that would be!
daxfohl 10y ago
By writing unit tests? Unit tests existed decades before they were popularized. Nobody "invented" them.
nickpsecurity 10y ago
Cacti had a nice summary. Some of their methods are also detailed in NASA's Software Safety Guidebook that was recently posted. My comment below tells you which pages have good stuff plus, as usual, has a link to the PDF itself. https://news.ycombinator.com/item?id=12016046 https://news.ycombinator.com/item?id=12016046 Here's a recent comment with so-called Correct-by-Construction methods for software development that knock out tons of defects. Usually don't have unit testing. The cost ranged from being lower (eg some Cleanroom projects) due to less debugging to around 50% higher (eg typical Altran/Praxis number). Time-to-market isn't significantly effected with some but is sacrificed for others. So, you don't need several hundred million in QA as some suggested. https://news.ycombinator.com/item?id=12081603 https://news.ycombinator.com/item?id=12081603
PaulKeeble 10y ago
I used to work in aerospace as a hardware/software engineer and while my team did use unit tests that wasn't how software was tested and qualified. The process in aerospace is vastly slower and that can be signified by the fact that on average a programmer doing aerospace produces just 1000 lines of code a year. There is a clear reason why aerospace engineers produce a lot less code: - Documentation - You get a pile as tall as your desk for a few thousand lines of code, the software is designed to the degree of knowing precisely what the maximum N can be going into a function and its exact runtime, every function has a fixed maximum and minimum values and a runtime associated with it preallocated before anyone writes code. Then the code is checked against the documentation. - Bench testing - We had a complete dev/test environment in which all in dev hardware/software could do missions and had all the different teams and their avionics device within a network interacting. You then flew countless "missions" (real cockpit parts but swivel chair + simulator) testing all the different scenarios and what you just did. - The wider team of engineers working on the avionics software would print out the entire code and all branches and "walk" the entire system from beginning to end with documentation in hand. Every line is validated by someone from a different team and every decision scrutinised. - System test - multiple phases of. Completely separate teams would test every new release to death rigorously, against the clients specs, against the documentation produced and also with their own independent simulator and cockput setup. Not only that but there was a second team doing the same thing trying to catch the first in missing something. - Then it spends years being tested in prototype vehicles before finally being signed off as ready. End to end it took about 10 years to get 60k lines of code released working full time on one avionics device. We had unit tests but only for the purpose of testing the hardware not really for our own software beyond startup tests. All of that is the rigour coming out of one principle, every time you find a problem in the code at any point its not the individuals fault its the teams fault and you work out the genuine cause and ensure it can't slip through again. Everything must be testable. There is quite a lot of parallels with unit testing and indeed it could be a better way to capture tests for aerospace, potentially a slightly less man intensive way as to run all the tests everytime a new release is produced, but it wouldn't look anything like what your typical commercial company does as to them its not worth reducing the 10 bugs per 1000 lines of code they currently have down to more or less 0. Unit testing in aerospace would be about efficient repetition of running known tests much as it is in the commercial space but not driving any form of design process, but I could see test definitions being produced from the function lists. We did a lot of specification testing and limiting of language to allow that to occur and make the code more straight forward.
twunde 10y ago
From http://link.springer.com/article/10.1007/BF01845743#page-1 http://link.springer.com/article/10.1007/BF01845743#page-1 1) Integration testing, 2)Systems testing, 3)load testing and 4) user acceptance testing. This would have been done with simulation tools alongside manual testing. Think about the way hardware testing is done and extrapolate those techniques to software.
- ubercode5 10y ago
  This is right on money. The test engineers as well as the implementation engineers also have a lot of experience within their specialized areas. This provides a lot of insight into expected results when running these different types of tests and scenarios. Also, these tests are long duration. Multiple iterations of simulations can take years to complete.
daxfohl 10y ago
The funny thing is it's not just "NASA way back when". Even now, the most critical code (e.g. implantable heart defibrillators) likely has less "unit test" than advert code that posts junk to your facebook wall. Waterfall is still the more sensible approach in some industries.
collyw 10y ago
According to this, unit tests are one of the least effective ways of catching bugs. https://kev.inburke.com/kevin/the-best-ways-to-find-bugs-in-your-code/ https://kev.inburke.com/kevin/the-best-ways-to-find-bugs-in-...
andersthue 10y ago
This is an article about the team writing code for the space shuttle, it is not so much about how they tested but more about the (insane) amount of time and energy spent on making sure everything worked. http://www.fastcompany.com/28121/they-write-right-stuff http://www.fastcompany.com/28121/they-write-right-stuff
abalashov 10y ago
Really? Because, how could quality assurance, software testing, and formal verification have existed before some hipster buzzword for a particular approach to it came about?
avalenciano 10y ago
Talk by Dr. Gerard Holzmann (leads the JPL Laboratory for Reliable Software): https://www.youtube.com/watch?v=x6yDubV3a9I https://www.youtube.com/watch?v=x6yDubV3a9I You can also search for the paper/final report: "NASA Study on Flight Software Complexity" to expand the topic.
mbrodersen 10y ago
To create really reliable software you either use formal methods (seL4 kernel, CompCert etc.) or you spend a huge amount of money and work really slow. NASA is spending a lot of money and working really slow. In other words, they are not in any way state-of-the-art. Just brute force.

PaulHoule 10y ago

Code review.

nekopa 10y ago

Process too.

hollander 10y ago

Back in the times when we counted in bytes or kilobytes, programs were simpler to manage. One person could overview all code. Now we have programs with million lines. Nobody can oversee that.

aardvark291 10y ago
"One person could overview all code." Not sure that's true. Imagine reviewing the code of the Apollo Guidance Computer, just to take an example from the 60's. The stack of paper containing the source code printout is more than five feet tall: http://skeptics.stackexchange.com/questions/31602/is-this-a-photo-of-margaret-hamilton-standing-next-to-apollo-project-code-that-s http://skeptics.stackexchange.com/questions/31602/is-this-a-...
- nostrademons 10y ago
  It's on GitHub: https://github.com/chrislgarry/Apollo-11 https://github.com/chrislgarry/Apollo-11 Reviewing it would be non-trivial, but in terms of absolute code size, it's not much larger than a moderately-popular open-source project today. It's significantly smaller than React, for example: https://github.com/facebook/react/tree/master/src https://github.com/facebook/react/tree/master/src Edit: Downloaded both repositories. 'wc' puts the AGC at roughly 60K LOC, while just the 'src' directory in React is 70K.

yeukhon 10y ago

My wild guess is just a lot of testing. Rigorous testing. Note a lot of early NASA code were assembly.

typetypetype 10y ago

I think you mean to say "automated, script-based unit tests". You can test code manually.

jpindar 10y ago
Back in the day, people used to do something called "desk checking".

efnx 10y ago

A while back someone posted some programming guidelines issued by nasa. It detailed requiring assert statements at least every 10 loc. It spoke of many of other things as well, but that was one that stuck in my head.

athenot 10y ago
For whatever it's worth, assert statements were also heavily used in embedded digital TV software. The philosophy is that things either work or they don't. With enough QA and field testing, those asserts will show up if state goes awry.
pkaye 10y ago
This is something I do myself. As I build up the code, I write tons of self checking asserts so it fails quite quickly on the first sign of something wrong. At some point, I'll remove the less likely asserts as they just clutter up the code.
jdavis703 10y ago
Is this it this: http://pixelscommander.com/wp-content/uploads/2014/12/P10.pdf http://pixelscommander.com/wp-content/uploads/2014/12/P10.pd...? If so it looks like functions can't be more than 60 lines (rule #4), and that the "assertion density" (rule #5) must be two asserts per function.

deleted 10y ago

[deleted]

geonnave 10y ago

Some thoughts I have: - I think they did apply methodologies of reliability that worked in hardware, to software (e.g redundancy, automatic fault detection/tracing/correction) - Also, in the early days, software was so tied to hardware, and hardware was faaar less complex, so they probably could understand everything that was happening in the system at a given time

noblethrasher 10y ago

This article by Walter Bright (creator of D, also hangs out on HN as 'WalterBright) might be illuminating as well as instructive: http://www.drdobbs.com/architecture-and-design/safe-systems-from-unreliable-parts/228701716 http://www.drdobbs.com/architecture-and-design/safe-systems-...

wwkeyboard 10y ago

In addition to everything else mentioned here they spent a TON of money[1]. For example they spent what would be 3.6 billion 2016 dollars on Guidance and Navigation. If your current project could drop a few hundred million on QC I bet quality would go way up regardless of automated testing or not. [1]http://history.nasa.gov/SP-4029/Apollo_18-16_Apollo_Program_Budget_Appropriations.htm http://history.nasa.gov/SP-4029/Apollo_18-16_Apollo_Program_...

kakali 10y ago
I think your number includes design and building one of the first solid state computers.
- wwkeyboard 10y ago
  The "few hundred million for quality control" number still stands. In addition they were able to draw on everyone from the people designing the transistors through to the programers to debug problems. Not something many modern projects can do.

abtinf 10y ago

Unit testing is not a method of achieving reliability, but of accelerating development. They do not prove that a system functions correctly, which is more in the domain of systems and integration testing. Also, testing is only one small part of building reliable systems. Instead, reliability requires actual engineering practice, which is non-existent in the software world except in a few life-critical domains. NASA and their contractor's software engineering practices have been covered extensively elsewhere. Here are some links. https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff http://history.nasa.gov/computers/Ch4-5.html http://history.nasa.gov/computers/Ch4-5.html

fokov 10y ago
Unit tests do not even prove that unit functions correctly either. Code coverage doesn't necessarily mean anything too.
rootusrootus 10y ago
My takeaway from reading the fastcompany.com article is that it's not process improvement per se that provides quality software, it's money. NASA has invested a great deal more time, and therefore money, into developing software than a typical commercial software company can afford. We will need quite a lot more programmers necessarily getting paid a lot less if we want commercially viable software production within spitting distance of NASA quality expectations.
- Spearchucker 10y ago
  That's just it. If a bug in software can cause death you're not going to use agile. You're going to use a waterfall-like process, and you're going to have stringent quality gates at every single transition - from envisioning, through requirements analysis, design, development, test, release, and into operation. It's the only way. I know because I built a military guaranteed messaging system. Tech aspect was small, but it took 24 months.
  - e12e 10y ago

ThomPete 10y ago

I think it's important to remember that the kind of software written for any kind of flight control or guidance system are very very very far from the kind of dynamic software normally done today. There where more likely to be hardware errors than software errors, thats how close to the hardware they wrote.

invaliduser 10y ago

Unit Tests are not the one and only metrics for software quality. Believe it or not, before unit tests were coined, we managed to build reliable software too. Actually a lot of the internet was actually built that way. But your assumption is very interesting, because it states a very common misconception of quality (whatever that means, reliability, maintainability, resilience, etc): * "Unit tests imply reliable software". Well, it does not, you'd need reliable unit tests for that, and from what I've seen in the industry, not every unit test out in the wild increases reliability. * "Without unit tests, you can't have quality software". That is just not true either. First, not all tests are unit tests, end-to-end (integration/functional/whatever) test are extremely useful and raises the quality too. Even without any automated test, the quality of documentation, contract programming, manual testing, etc also help a lot. Also, note that apart from testing, a lot of different practices contribute to reliability. Separation of concerns, the language level of abstraction, the overall experience and involvements of the developers, and many others.

jeremiep 10y ago
I find that the people who are the most obsessed about unit testing generally aren't good programmers to begin with. To me, keeping code simple and easy to reason about is what yields reliable software. If you can't wrap your head around something in the first place then tests are only going to give you the illusion of reliability. Its as Rich Hickey put it: "What does every bug have in common? They all passed the type checker and the tests!"
- Cacti 10y ago
  As the saying goes, crap in, crap out. It's just as easy to write crappy unit tests as it is crappy code.
  - tmm 10y ago
    Or, as W. E. Deming wrote: "You cannot inspect quality into a product."
  - finishingmove 10y ago

tuned 10y ago

Asserting about involved variables each 10 lines of code (:

carapace 10y ago

Margaret Hamilton

nickpsecurity 10y ago
I suggest always including links when you drop her name as significance will be lost on many. https://en.wikipedia.org/wiki/Margaret_Hamilton_(scientist) https://en.wikipedia.org/wiki/Margaret_Hamilton_(scientist) http://htius.com/ http://htius.com/
- carapace 10y ago
  I agree, I was trying to be "arch" ;-) Here's a good synopsis: http://www.history.nasa.gov/alsj/a11/a11Hamilton.html http://www.history.nasa.gov/alsj/a11/a11Hamilton.html """ NASA Press Release, 3 September 2003 NASA HONORS APOLLO ENGINEER Margaret Hamilton, leader of the team that developed the flight software for the agency's Apollo missions, has been granted a NASA Exceptional Space Act Award for her scientific and technical contributions. "The Apollo flight software Ms. Hamilton and her team developed was truly a pioneering effort," said NASA Administrator Sean O'Keefe. "The concepts she and her team created became the building blocks for modern 'software engineering.' It's an honor to recognize Ms. Hamilton for her extraordinary contributions to NASA," he said. Dr. Paul Curto, senior technologist for NASA's Inventions and Contributions Board nominated Hamilton for the award. Curto said, "I was surprised to discover she was never formally recognized for her groundbreaking work. Her concepts of asynchronous software, priority scheduling, end-to-end testing, and man-in-the-loop decision capability, such as priority displays, became the foundation for ultra-reliable software design." One example of the value of Hamilton's software work occurred during the Apollo 11 mission. Approximately three minutes before Eagle's touchdown on the moon, the software over rode a command to switch the flight computer's priority processing to a radar system whose 'on' switch had been manually activated due to a faulty written operations script provided to the crew. The action by the software permitted the mission to safely continue. """
- sebastianconcpt 10y ago
  Indeed. Thank you for the links

dsl 10y ago

Your submission falsely implies that unit tests create reliable software. But that aside... Avionics software is subjected to multiple rounds of peer review (often by different companies), integration testing, and in some cases multiple different implementations are developed and black box tested against each other for different outputs given the same inputs.

owiejg 10y ago

I think they spend a lot of time doing very detailed code reviews.

njharman 10y ago

Unit tests aren't really about testing (in the validating sense) software. They in fact are quite deficient in that role. No integration tests, no end-to-end tests, no fuzz testing, etc. Unit Tests are foremost a software development tool. They force (in order to be unit testable) separation of concerns, definition of interfaces (i.e. the "contract" with function's users), etc. Mechanically, they also enable validating that internal refactoring has not violated that contract.

gwbas1c 10y ago
As big of a fan as I am about unit tests, I find their value somewhat dubious. They take a long time to write, and when they fail, it often means the unit test needs to be updated. Mocking interactions among objects takes a long time. What I find has a better ROI is using the same unit testing tools, but writing what unit test purists will call integration tests. Basically, using unit test tools where most of a program is instantiated, but I/O is mocked, is much more useful. When these tests fail, it usually indicates a bug instead of a part of a unit test that needs to be updated.
- flukus 10y ago
  It sounds like a problem with your tests actually. Unit test shouldn't take a long time to write and the should only fail for a very specific.
- njharman 10y ago
  This is why/how they are a development tool. "Mocking interactions among objects", often breaking, taking too long to write (I expect to spend 30% dev on units, 30% on code rest thinking, planning, etc). Those are all indicators that your code/architecture is bad. Too complex, too coupled, too much concern, not layered into clean interfaces, etc.

daxfohl 10y ago

Who invented unit tests? Does that person have a monopoly on highly reliable software? What an awesome patent trolling opportunity that would be!

daxfohl 10y ago

By writing unit tests? Unit tests existed decades before they were popularized. Nobody "invented" them.

nickpsecurity 10y ago

Cacti had a nice summary. Some of their methods are also detailed in NASA's Software Safety Guidebook that was recently posted. My comment below tells you which pages have good stuff plus, as usual, has a link to the PDF itself. https://news.ycombinator.com/item?id=12016046 https://news.ycombinator.com/item?id=12016046 Here's a recent comment with so-called Correct-by-Construction methods for software development that knock out tons of defects. Usually don't have unit testing. The cost ranged from being lower (eg some Cleanroom projects) due to less debugging to around 50% higher (eg typical Altran/Praxis number). Time-to-market isn't significantly effected with some but is sacrificed for others. So, you don't need several hundred million in QA as some suggested. https://news.ycombinator.com/item?id=12081603 https://news.ycombinator.com/item?id=12081603

PaulKeeble 10y ago

I used to work in aerospace as a hardware/software engineer and while my team did use unit tests that wasn't how software was tested and qualified. The process in aerospace is vastly slower and that can be signified by the fact that on average a programmer doing aerospace produces just 1000 lines of code a year. There is a clear reason why aerospace engineers produce a lot less code: - Documentation - You get a pile as tall as your desk for a few thousand lines of code, the software is designed to the degree of knowing precisely what the maximum N can be going into a function and its exact runtime, every function has a fixed maximum and minimum values and a runtime associated with it preallocated before anyone writes code. Then the code is checked against the documentation. - Bench testing - We had a complete dev/test environment in which all in dev hardware/software could do missions and had all the different teams and their avionics device within a network interacting. You then flew countless "missions" (real cockpit parts but swivel chair + simulator) testing all the different scenarios and what you just did. - The wider team of engineers working on the avionics software would print out the entire code and all branches and "walk" the entire system from beginning to end with documentation in hand. Every line is validated by someone from a different team and every decision scrutinised. - System test - multiple phases of. Completely separate teams would test every new release to death rigorously, against the clients specs, against the documentation produced and also with their own independent simulator and cockput setup. Not only that but there was a second team doing the same thing trying to catch the first in missing something. - Then it spends years being tested in prototype vehicles before finally being signed off as ready. End to end it took about 10 years to get 60k lines of code released working full time on one avionics device. We had unit tests but only for the purpose of testing the hardware not really for our own software beyond startup tests. All of that is the rigour coming out of one principle, every time you find a problem in the code at any point its not the individuals fault its the teams fault and you work out the genuine cause and ensure it can't slip through again. Everything must be testable. There is quite a lot of parallels with unit testing and indeed it could be a better way to capture tests for aerospace, potentially a slightly less man intensive way as to run all the tests everytime a new release is produced, but it wouldn't look anything like what your typical commercial company does as to them its not worth reducing the 10 bugs per 1000 lines of code they currently have down to more or less 0. Unit testing in aerospace would be about efficient repetition of running known tests much as it is in the commercial space but not driving any form of design process, but I could see test definitions being produced from the function lists. We did a lot of specification testing and limiting of language to allow that to occur and make the code more straight forward.

twunde 10y ago

From http://link.springer.com/article/10.1007/BF01845743#page-1 http://link.springer.com/article/10.1007/BF01845743#page-1 1) Integration testing, 2)Systems testing, 3)load testing and 4) user acceptance testing. This would have been done with simulation tools alongside manual testing. Think about the way hardware testing is done and extrapolate those techniques to software.

ubercode5 10y ago
This is right on money. The test engineers as well as the implementation engineers also have a lot of experience within their specialized areas. This provides a lot of insight into expected results when running these different types of tests and scenarios. Also, these tests are long duration. Multiple iterations of simulations can take years to complete.

daxfohl 10y ago

The funny thing is it's not just "NASA way back when". Even now, the most critical code (e.g. implantable heart defibrillators) likely has less "unit test" than advert code that posts junk to your facebook wall. Waterfall is still the more sensible approach in some industries.

collyw 10y ago

According to this, unit tests are one of the least effective ways of catching bugs. https://kev.inburke.com/kevin/the-best-ways-to-find-bugs-in-your-code/ https://kev.inburke.com/kevin/the-best-ways-to-find-bugs-in-...

andersthue 10y ago

This is an article about the team writing code for the space shuttle, it is not so much about how they tested but more about the (insane) amount of time and energy spent on making sure everything worked. http://www.fastcompany.com/28121/they-write-right-stuff http://www.fastcompany.com/28121/they-write-right-stuff

abalashov 10y ago

Really? Because, how could quality assurance, software testing, and formal verification have existed before some hipster buzzword for a particular approach to it came about?

avalenciano 10y ago

Talk by Dr. Gerard Holzmann (leads the JPL Laboratory for Reliable Software): https://www.youtube.com/watch?v=x6yDubV3a9I https://www.youtube.com/watch?v=x6yDubV3a9I You can also search for the paper/final report: "NASA Study on Flight Software Complexity" to expand the topic.

mbrodersen 10y ago

To create really reliable software you either use formal methods (seL4 kernel, CompCert etc.) or you spend a huge amount of money and work really slow. NASA is spending a lot of money and working really slow. In other words, they are not in any way state-of-the-art. Just brute force.