6 ms·

Here's a paper[1] for a space conference in Montreal written a few years ago on DO-178 standards and applying the FAA's certification process to space systems d

Here's a paper[1] for a space conference in Montreal written a few years ago on DO-178 standards and applying the FAA's certification process to space systems development (DO-178 sets the bar for "testing the shit" out of software, I've seen projects held up for years because it's like approaching a limit f(n) -> L (where f(n) is the software development cycle and L is getting a non-bought DER to sign off on it) proving your system to its requirements). [1] http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?2013ESASP.715E..63D&data_type=PDF_HIGH&whole_paper=YES&type=PRINTER&filetype=.pdf http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_quer...

nickpsecurity 9y ago
DO-178B was one of best things to happen in software assurance. It was a non-prescriptive method of certification that forced precise requirements, design, and code with reviews of each and tracability between them. The high cost of certification and certification failure led to a booming ecosystem of supporting tools (esp static analysis or runtimes) and pre-made components (esp RTOS's & middleware). Some companies even specialized in handling the administrative overhead. The result was that certified systems usually had very, high quality. Meanwhile, there were people arguing on HN and other places about whether a certification could ever be useful for software. Anyone reading the above paper should skip to the summary to get a high-level view of the overhead and resulting benefits. The author of the paper agrees with me that the certification is beneficial. The gist is you keep things simple, think about everything in lifecycle, test everything in lifecycle, throw every analysis tool you can at the design or code, esp take your time to develop the thing, and document all of this in a way where any aspect is easy to verify by a 3rd party. That's how high-assurance systems are done. They usually work, too. Some work for a long, long, long time. :)
- mcguire 9y ago
  DO-178C has a formal methods follow-on. I haven't seen it, though.
- ptero 9y ago
  Applied very narrowly to the right problem, maybe. But I have also seen DO-178B to lead to a completely non-maintainable mess. That mess helps trace each statement to a requirement, but leads to code that is unreadable to anyone used to normal code. It is not that "no extra code" and exactly meeting requirements is a bad model in itself. However writing code in a way that makes it convenient to prove the above encourages unnatural coding styles and programmatic contortions. YMMV however.
  - alexanderdmitri 9y ago
    This happens a lot. IMO, the best method to mitigate this is to design and write your code base to the system requirements rather than try to perfect the low level requirements before writing any code. You can then reverse-engineer the low level requirements from source, deriving your verbiage from the system requirements and logic from the actual code. This way your low-level/unit testing is precise and verifiable while also ensuring strong traceability up and down the chain. If there ends up being low level requirements/code that don't trace up to the system for some reason, then either there's been a disconnect between design and execution or, if not, you make the case for a derived requirement.
    - w_t_payne 9y ago
      I agree that the left hand side of the Vee has to be seen as a two way street. As we develop the code we gain experience and insights which enable us to elaborate and refine our requirements. The requirements then act both to guide the development of the system and also to document the lessons that have been learned during development. The output of early requirements elaboration phases are then draft, rather than final, requirements sets. The key, for me, is the tooling that we have around maintaining requirements and the traces between them and the code. I think that it should be possible to navigate traces and update (low level) requirements and test specifications without leaving the text editor. If it takes 5 minutes to open DOORS and to find the relevant requirement, then you aren't likely to keep it as up to date as you should. The same is true if the requirements are stored on a spreadsheet in a document management system. Some ALM/IDE tools may enable you to manage things in a better way. From what I have seen, mbeddr is a really interesting experiment along these lines. I also suspect that Visual Studio and Eclipse have some pretty powerful features in this regard. In my side project (and purely for my own 'entertainment') I am experimenting with ways of embedding low level requirements into comments in the code (the build extracts them and can update DOORS or some other tool), as well as a generic trace system that allows different classes of traceable item to be defined and indexed. The build can enforce requirements coverage in the same way that it enforces test coverage, and because the text and metadata of requirements is easily accessible to the build, we can use NLP tools to enforce the use of restricted natural language or DSLs in the requirements text. The ultimate goal being to use machine learning to learn the correlation between requirements changes and code changes so that a change in one can help generate advice and guidance on what needs to change in the other.

nickpsecurity 9y ago

DO-178B was one of best things to happen in software assurance. It was a non-prescriptive method of certification that forced precise requirements, design, and code with reviews of each and tracability between them. The high cost of certification and certification failure led to a booming ecosystem of supporting tools (esp static analysis or runtimes) and pre-made components (esp RTOS's & middleware). Some companies even specialized in handling the administrative overhead. The result was that certified systems usually had very, high quality. Meanwhile, there were people arguing on HN and other places about whether a certification could ever be useful for software. Anyone reading the above paper should skip to the summary to get a high-level view of the overhead and resulting benefits. The author of the paper agrees with me that the certification is beneficial. The gist is you keep things simple, think about everything in lifecycle, test everything in lifecycle, throw every analysis tool you can at the design or code, esp take your time to develop the thing, and document all of this in a way where any aspect is easy to verify by a 3rd party. That's how high-assurance systems are done. They usually work, too. Some work for a long, long, long time. :)

mcguire 9y ago
DO-178C has a formal methods follow-on. I haven't seen it, though.
ptero 9y ago
Applied very narrowly to the right problem, maybe. But I have also seen DO-178B to lead to a completely non-maintainable mess. That mess helps trace each statement to a requirement, but leads to code that is unreadable to anyone used to normal code. It is not that "no extra code" and exactly meeting requirements is a bad model in itself. However writing code in a way that makes it convenient to prove the above encourages unnatural coding styles and programmatic contortions. YMMV however.
- alexanderdmitri 9y ago
  This happens a lot. IMO, the best method to mitigate this is to design and write your code base to the system requirements rather than try to perfect the low level requirements before writing any code. You can then reverse-engineer the low level requirements from source, deriving your verbiage from the system requirements and logic from the actual code. This way your low-level/unit testing is precise and verifiable while also ensuring strong traceability up and down the chain. If there ends up being low level requirements/code that don't trace up to the system for some reason, then either there's been a disconnect between design and execution or, if not, you make the case for a derived requirement.
  - w_t_payne 9y ago
    I agree that the left hand side of the Vee has to be seen as a two way street. As we develop the code we gain experience and insights which enable us to elaborate and refine our requirements. The requirements then act both to guide the development of the system and also to document the lessons that have been learned during development. The output of early requirements elaboration phases are then draft, rather than final, requirements sets. The key, for me, is the tooling that we have around maintaining requirements and the traces between them and the code. I think that it should be possible to navigate traces and update (low level) requirements and test specifications without leaving the text editor. If it takes 5 minutes to open DOORS and to find the relevant requirement, then you aren't likely to keep it as up to date as you should. The same is true if the requirements are stored on a spreadsheet in a document management system. Some ALM/IDE tools may enable you to manage things in a better way. From what I have seen, mbeddr is a really interesting experiment along these lines. I also suspect that Visual Studio and Eclipse have some pretty powerful features in this regard. In my side project (and purely for my own 'entertainment') I am experimenting with ways of embedding low level requirements into comments in the code (the build extracts them and can update DOORS or some other tool), as well as a generic trace system that allows different classes of traceable item to be defined and indexed. The build can enforce requirements coverage in the same way that it enforces test coverage, and because the text and metadata of requirements is easily accessible to the build, we can use NLP tools to enforce the use of restricted natural language or DSLs in the requirements text. The ultimate goal being to use machine learning to learn the correlation between requirements changes and code changes so that a change in one can help generate advice and guidance on what needs to change in the other.