7 ms·

Rules for Writing Safety Critical Code

61 points by plumeria 11y ago

jacquesm 11y ago
In 'C' or 'C++'. Though some of the rules are portable to other languages. I'd add: have someone else familiar with the code base look over your code, add tests for your functions, test exhaustively where feasible, document your code and document your reasoning behind the code as well, make it look good, try to avoid being clever.
- AtmaScout 11y ago
  I agree completely with avoid being clever. If the situation requires you write clever code, make sure it is commented fully.
  - mannykannot 11y ago
    Where 'fully commented' includes your arguments for its correctness. A person who is writing 'clever' code that is not backed by an argument for its correctness is not being clever: the best he can hope for is lucky.
SCHiM 11y ago
However from the top of my head I can give you one example that is likely used in very security-critical applications that also uses goto: The linux kernel. The kernel coding conventions actually state that this is because over-zealous nesting is more harmful than using goto statements. >The rationale for using gotos is: - unconditional statements are easier to understand and follow - nesting is reduced - errors by not updating individual exit points when making modifications are prevented - saves the compiler work to optimize redundant code away ;) https://www.kernel.org/doc/Documentation/CodingStyle https://www.kernel.org/doc/Documentation/CodingStyle
- TickleSteve 11y ago
  security-critical != safety-critical. Linux would never get chosen for any higher level safety-related s/w.
  - PinguTS 11y ago
    Safety critical and security critical share a lot of goals like: sanity checking of input data, buffer overflows, …
  - fit2rule 11y ago
    Linux is deployed all over the world in safety-critical applications .. if you take a train in Europe, parts of China, Brazil, some parts of the US .. Linux is running along, track-side, protectin' yo' ass.. Linux is used in systems that have SIL-4 and ISO 27001 up the wazoo, its just that in the commercial/industrial use, 'tis all extremely well documented. And tested. Tested, tested, tested. Onboard, offline, staged, etc. Test the hell out of all safety-critical software. No goto un-tested. No branches not taken.
    - TickleSteve 11y ago
      In those systems, the Linux kernel tends to be compartmentalised, basically being run as a task under the control of the real RTOS while the safety critical & hard real-time portions are run in a proper RTOS. The other situation is that the actual control is deferred to a proper RTOS/micro combination which can do the correct thing when Linux goes off the rails. The Linux kernel is not a hard-real-time system, it doesn't have complete test coverage and its inclusion in highly-safety-critical systems is as a lower-safety-level subsystem. You will not find a Linux task controlling (for example) a GPIO enabling an X-Ray beam in human rated medical devices. BTW: I can promise you that there are many untested goto's in the Linux Kernel...
      - deleted 11y ago
        [deleted]
      - fit2rule 11y ago
        "proper" RTOS. What part of Linux precludes it from being a proper RT OS in the hands of industrial developers whose purpose it is to certify it as such? >BTW: I can promise you that there are many untested goto's in the Linux Kernel... Not in a development environment looking for SIL-4 certification, there isn't. Perhaps in the public sphere you may feel that is an adequate position, but in industrial computing a significant part of the value is in the testing and real certification of the Linux kernel for certain ratings. >The Linux kernel is not a hard-real-time system, it doesn't have complete test coverage and its inclusion in highly-safety-critical systems is as a lower-safety-level subsystem I assure you, I have found none of these statements to be truly true.
- erikb 11y ago
  That's pretty much my point. By avoiding to use some tools completely one does not necessarily increase the stability of software. I just hope that such kind of rules allow well documented/argumented exceptions. But I guess even in safety-critical environments you will have to handle exceptions (small, like in "we did something different than what was agreed on" not as in "throw BufferOverflowException") anyway. No system is perfect.
TickleSteve 11y ago
A lot of these rules are just good practice for general embedded s/w development (see also MISRA), such as fixed upper bounds on loops, etc are intended to minimise dynamic situations and assist analysis. Just to be clear... Following these rules does not get you any kind of safety related certification. There is a lot more work, both up-front in requirements & analysis, and after implementation to gain any kind of certification.
- PinguTS 11y ago
  Following some or all of those rules would be wise to be used by any developer, especially Web developers. That would eliminate a number of bugs. Bugs, that seems get fixed over and over again.
  - TickleSteve 11y ago
    exactly... higher level s/w has a lot to learn from embedded techniques regarding performance & reliability.
    - eterm 11y ago
      At a higher level, especially in web development, readability is often more important than performance & reliability. Time to market and how easy it is for a new team member to pick up the code is more important than catching every bug or sub-par performance because patching and deployment is relatively cheap.
      - falcolas 11y ago
        Following these rules will improve the code readability as a side effect of being safer - all of the constructs will be simpler, which makes the constructs easier to understand.
      - TickleSteve 11y ago
        readbility (and more importantly, understandability) is critical in these systems, this is one of the things that higher level languages can learn. Higher level s/w has piled on the abstraction layers and that hides what is really happening on the h/w level. Web s/w has gone in direction of simplicity-by-increasing-abstraction. Embedded s/w gains simplicity by removing it. Of course, some knowledge of the h/w is necessary for this, but thats a prerequisite to performant systems anyway.
chopin 11y ago
I am not sure whether I understand rule #2 correctly. My main use case for loops is iterating over an array or a list of statically unknown size, therefore the loop is limited to the actual size of the array or list. Would that count as statically provable? At least in Java, these cannot hold more than 2^31 elements for the standard library collections. For this use case, would it be required to have a hard-coded upper bound additionally?
- gte525u 11y ago
  It's closely coupled with rule #3. An array at least in the safety critical embedded software have a fixed size determined at initialization. You may have a sentinel value you're looking for, but, the loop should never exceed the preallocated size. There are another set of arguments /against/ using stopping the loop at the sentinel values because it may make the software less deterministic. But, that's another topic.
  - erikb 11y ago
    Less deterministic? Isn't it quite deterministic if something stops when a static limit is exceeded?
    - victorNicollet 11y ago
      This helps ensure that if your software runs in 10ms on the test data set, then it will always run in 10ms on all data sets. In general, it is recommended to avoid "best case" optimization, so that your critical software is always running with "worst case" performance. EDIT: On re-reading your message and the message it is responding to, I believe you are meaning opposite things by "sentinel".
  - TickleSteve 11y ago
    An example wold be to use memcpy() instead of strcpy(). memcpy() always copies n bytes, regardless of actual string size so is very deterministic and (commonly) inlined so is static. strcpy() on the other hand would loop infinitely until it found a null-terminator, it also is not deterministic. memcpy() is easily analysable. strcpy() isn't.
lordnacho 11y ago
What about using smart pointers? There's a whole class of potential errors related to using raw pointers.
- TickleSteve 11y ago
  smart-pointers are useful for automatic resource tracking. In embedded systems, very, very little is dynamically allocated. Everything is as static as possible. no new/delete malloc/free, etc. Its all static data. In general, embedded systems have taken the opposite tack of solving the memory allocation issues of malloc/free... we just don't do it. Desktop & server type applications use garbage collection/ARC type tracking. you wont find anything like that on small systems and certainly not in safety-critical code.
  - lordnacho 11y ago
    Good point, I forgot the context. Very similar to certain trading systems, where you also simply pre-allocate everything and avoid potential latency issues, along with giving fine grained control.
erikb 11y ago
I never worked in a safety critical environment, so my ideas are probably flawed. My feeling says that limiting loops and not using some language constructs can enhance quality that much. Checking and rechecking requirements, having different levels of testing, and most importantly have static and unit test checks on the source code abstraction level has a huge impact, though. Example: Even if you use only simple constructs like if and for you still run into problems with complexity because some problems are complex. Now the complexity simply is spread, which in some regards might make it easier to handle but in other cases makes it much harder to see the dependencies.
- TickleSteve 11y ago
  You are correct in those points, and implementation rules like these do not gain you a safety accreditation. There is a lot of upfront analysis of the system as a whole and the algorithms before a line of code is written. Complexity and dynamism is minimised and all code paths are checked in the final code. This is one of the few places where 100% code-coverage is needed. you need to account for every path in the code, all the way down to bare-metal. Also, a common thing to so is to have h/w fail-safe systems (iff possible), due to the difficulty in getting s/w approved.
lectrick 11y ago
This applies to machine-level C code and no high-level languages. Functional high-level languages prevent most of these out of the box, for example
- sacado2 11y ago
  This is far from true. > 1 Restrict to simple control flow constructs. Details give the precision : no recursion allowed. Functional languages are out, there. > 2 Give all loops a fixed upper-bound. Func languages give no warranty on that either. > 3 Do not use dynamic memory allocation after initialization. You have no access to memory layout, but func languages heavily rely on the creation on-the-fly of data and rely on garbage collection (which is forbidden, too). > 4 Limit functions to no more than 60 lines of text. > 5 Use minimally two assertions per function on average. > 6 Declare data objects at the smallest possible level of scope. > 7 Check the return value of non-void functions, and check the validity of function parameters. OK, that might be easier with func languages. > 8 Limit the use of the preprocessor to file inclusion and simple macros. Not relevant > 9 Limit the use of pointers. Use no more than two levels of dereferencing per expression. OK, no pointer manipulation with these languages. > 10 Compile with all warnings enabled, and use one or more source code analyzers. Name one professional code analyzer for any func language of your choice.
  - lectrick 11y ago
    > Details give the precision : no recursion allowed. Functional languages are out, there. > Func languages give no warranty on that either. Actually, what you can do in this case is when you call recursively, pass back an incremented counter variable as part of the state, and if it passes some upper limit, handle it appropriately. You don't need to "avoid recursion" just to get control over the number of times something loops. > and rely on garbage collection (which is forbidden, too) Fair enough, but if I work in a language that doesn't even give me much control (if any) over the GC, then how is this relevant to non-C coders, again? Which was my original point. > Name one professional code analyzer for any func language of your choice. Erlang (and by association Elixir) has Dialyzer http://www.erlang.org/doc/man/dialyzer.html http://www.erlang.org/doc/man/dialyzer.html which is a static code analyzer. Static code analysis has been endorsed by John Carmack (and of course, all the 3 year old links to the blog post are now dead, sigh) For anyone (like me) who is hooked on Elixir, https://github.com/jeremyjh/dialyxir https://github.com/jeremyjh/dialyxir provides a bit of easier Dialyzer management Elixir (and I suppose Erlang) has this concept of "specs" which you don't have to use, but which help code analysis if you do... http://elixir-lang.org/docs/v1.0/elixir/Kernel.Typespec.html http://elixir-lang.org/docs/v1.0/elixir/Kernel.Typespec.html
kazinator 11y ago
This is misleadingly titled; it should be called: Rules, Mostly Pertaining to C, For Writing Safety-Critical Code (with Rule 0 Removed) (Rule 0 is: Don't use C for safety-critical code!) (And you know there was a Rule [0], obviously!)

61 points by plumeria 11y ago

In 'C' or 'C++'. Though some of the rules are portable to other languages. I'd add: have someone else familiar with the code base look over your code, add tests for your functions, test exhaustively where feasible, document your code and document your reasoning behind the code as well, make it look good, try to avoid being clever.

AtmaScout 11y ago
I agree completely with avoid being clever. If the situation requires you write clever code, make sure it is commented fully.
- mannykannot 11y ago
  Where 'fully commented' includes your arguments for its correctness. A person who is writing 'clever' code that is not backed by an argument for its correctness is not being clever: the best he can hope for is lucky.

SCHiM 11y ago

However from the top of my head I can give you one example that is likely used in very security-critical applications that also uses goto: The linux kernel. The kernel coding conventions actually state that this is because over-zealous nesting is more harmful than using goto statements. >The rationale for using gotos is: - unconditional statements are easier to understand and follow - nesting is reduced - errors by not updating individual exit points when making modifications are prevented - saves the compiler work to optimize redundant code away ;) https://www.kernel.org/doc/Documentation/CodingStyle https://www.kernel.org/doc/Documentation/CodingStyle

TickleSteve 11y ago
security-critical != safety-critical. Linux would never get chosen for any higher level safety-related s/w.
- PinguTS 11y ago
  Safety critical and security critical share a lot of goals like: sanity checking of input data, buffer overflows, …
- fit2rule 11y ago
  Linux is deployed all over the world in safety-critical applications .. if you take a train in Europe, parts of China, Brazil, some parts of the US .. Linux is running along, track-side, protectin' yo' ass.. Linux is used in systems that have SIL-4 and ISO 27001 up the wazoo, its just that in the commercial/industrial use, 'tis all extremely well documented. And tested. Tested, tested, tested. Onboard, offline, staged, etc. Test the hell out of all safety-critical software. No goto un-tested. No branches not taken.
  - TickleSteve 11y ago
    In those systems, the Linux kernel tends to be compartmentalised, basically being run as a task under the control of the real RTOS while the safety critical & hard real-time portions are run in a proper RTOS. The other situation is that the actual control is deferred to a proper RTOS/micro combination which can do the correct thing when Linux goes off the rails. The Linux kernel is not a hard-real-time system, it doesn't have complete test coverage and its inclusion in highly-safety-critical systems is as a lower-safety-level subsystem. You will not find a Linux task controlling (for example) a GPIO enabling an X-Ray beam in human rated medical devices. BTW: I can promise you that there are many untested goto's in the Linux Kernel...
    - deleted 11y ago
      [deleted]
    - fit2rule 11y ago
      "proper" RTOS. What part of Linux precludes it from being a proper RT OS in the hands of industrial developers whose purpose it is to certify it as such? >BTW: I can promise you that there are many untested goto's in the Linux Kernel... Not in a development environment looking for SIL-4 certification, there isn't. Perhaps in the public sphere you may feel that is an adequate position, but in industrial computing a significant part of the value is in the testing and real certification of the Linux kernel for certain ratings. >The Linux kernel is not a hard-real-time system, it doesn't have complete test coverage and its inclusion in highly-safety-critical systems is as a lower-safety-level subsystem I assure you, I have found none of these statements to be truly true.
erikb 11y ago
That's pretty much my point. By avoiding to use some tools completely one does not necessarily increase the stability of software. I just hope that such kind of rules allow well documented/argumented exceptions. But I guess even in safety-critical environments you will have to handle exceptions (small, like in "we did something different than what was agreed on" not as in "throw BufferOverflowException") anyway. No system is perfect.

TickleSteve 11y ago

A lot of these rules are just good practice for general embedded s/w development (see also MISRA), such as fixed upper bounds on loops, etc are intended to minimise dynamic situations and assist analysis. Just to be clear... Following these rules does not get you any kind of safety related certification. There is a lot more work, both up-front in requirements & analysis, and after implementation to gain any kind of certification.

PinguTS 11y ago
Following some or all of those rules would be wise to be used by any developer, especially Web developers. That would eliminate a number of bugs. Bugs, that seems get fixed over and over again.
- TickleSteve 11y ago
  exactly... higher level s/w has a lot to learn from embedded techniques regarding performance & reliability.
  - eterm 11y ago
    At a higher level, especially in web development, readability is often more important than performance & reliability. Time to market and how easy it is for a new team member to pick up the code is more important than catching every bug or sub-par performance because patching and deployment is relatively cheap.
    - falcolas 11y ago
      Following these rules will improve the code readability as a side effect of being safer - all of the constructs will be simpler, which makes the constructs easier to understand.
    - TickleSteve 11y ago
      readbility (and more importantly, understandability) is critical in these systems, this is one of the things that higher level languages can learn. Higher level s/w has piled on the abstraction layers and that hides what is really happening on the h/w level. Web s/w has gone in direction of simplicity-by-increasing-abstraction. Embedded s/w gains simplicity by removing it. Of course, some knowledge of the h/w is necessary for this, but thats a prerequisite to performant systems anyway.

chopin 11y ago

I am not sure whether I understand rule #2 correctly. My main use case for loops is iterating over an array or a list of statically unknown size, therefore the loop is limited to the actual size of the array or list. Would that count as statically provable? At least in Java, these cannot hold more than 2^31 elements for the standard library collections. For this use case, would it be required to have a hard-coded upper bound additionally?

gte525u 11y ago
It's closely coupled with rule #3. An array at least in the safety critical embedded software have a fixed size determined at initialization. You may have a sentinel value you're looking for, but, the loop should never exceed the preallocated size. There are another set of arguments /against/ using stopping the loop at the sentinel values because it may make the software less deterministic. But, that's another topic.
- erikb 11y ago
  Less deterministic? Isn't it quite deterministic if something stops when a static limit is exceeded?
  - victorNicollet 11y ago
    This helps ensure that if your software runs in 10ms on the test data set, then it will always run in 10ms on all data sets. In general, it is recommended to avoid "best case" optimization, so that your critical software is always running with "worst case" performance. EDIT: On re-reading your message and the message it is responding to, I believe you are meaning opposite things by "sentinel".
- TickleSteve 11y ago
  An example wold be to use memcpy() instead of strcpy(). memcpy() always copies n bytes, regardless of actual string size so is very deterministic and (commonly) inlined so is static. strcpy() on the other hand would loop infinitely until it found a null-terminator, it also is not deterministic. memcpy() is easily analysable. strcpy() isn't.

lordnacho 11y ago

What about using smart pointers? There's a whole class of potential errors related to using raw pointers.

TickleSteve 11y ago
smart-pointers are useful for automatic resource tracking. In embedded systems, very, very little is dynamically allocated. Everything is as static as possible. no new/delete malloc/free, etc. Its all static data. In general, embedded systems have taken the opposite tack of solving the memory allocation issues of malloc/free... we just don't do it. Desktop & server type applications use garbage collection/ARC type tracking. you wont find anything like that on small systems and certainly not in safety-critical code.
- lordnacho 11y ago
  Good point, I forgot the context. Very similar to certain trading systems, where you also simply pre-allocate everything and avoid potential latency issues, along with giving fine grained control.

erikb 11y ago

I never worked in a safety critical environment, so my ideas are probably flawed. My feeling says that limiting loops and not using some language constructs can enhance quality that much. Checking and rechecking requirements, having different levels of testing, and most importantly have static and unit test checks on the source code abstraction level has a huge impact, though. Example: Even if you use only simple constructs like if and for you still run into problems with complexity because some problems are complex. Now the complexity simply is spread, which in some regards might make it easier to handle but in other cases makes it much harder to see the dependencies.

TickleSteve 11y ago
You are correct in those points, and implementation rules like these do not gain you a safety accreditation. There is a lot of upfront analysis of the system as a whole and the algorithms before a line of code is written. Complexity and dynamism is minimised and all code paths are checked in the final code. This is one of the few places where 100% code-coverage is needed. you need to account for every path in the code, all the way down to bare-metal. Also, a common thing to so is to have h/w fail-safe systems (iff possible), due to the difficulty in getting s/w approved.

lectrick 11y ago

This applies to machine-level C code and no high-level languages. Functional high-level languages prevent most of these out of the box, for example

sacado2 11y ago
This is far from true. > 1 Restrict to simple control flow constructs. Details give the precision : no recursion allowed. Functional languages are out, there. > 2 Give all loops a fixed upper-bound. Func languages give no warranty on that either. > 3 Do not use dynamic memory allocation after initialization. You have no access to memory layout, but func languages heavily rely on the creation on-the-fly of data and rely on garbage collection (which is forbidden, too). > 4 Limit functions to no more than 60 lines of text. > 5 Use minimally two assertions per function on average. > 6 Declare data objects at the smallest possible level of scope. > 7 Check the return value of non-void functions, and check the validity of function parameters. OK, that might be easier with func languages. > 8 Limit the use of the preprocessor to file inclusion and simple macros. Not relevant > 9 Limit the use of pointers. Use no more than two levels of dereferencing per expression. OK, no pointer manipulation with these languages. > 10 Compile with all warnings enabled, and use one or more source code analyzers. Name one professional code analyzer for any func language of your choice.
- lectrick 11y ago
  > Details give the precision : no recursion allowed. Functional languages are out, there. > Func languages give no warranty on that either. Actually, what you can do in this case is when you call recursively, pass back an incremented counter variable as part of the state, and if it passes some upper limit, handle it appropriately. You don't need to "avoid recursion" just to get control over the number of times something loops. > and rely on garbage collection (which is forbidden, too) Fair enough, but if I work in a language that doesn't even give me much control (if any) over the GC, then how is this relevant to non-C coders, again? Which was my original point. > Name one professional code analyzer for any func language of your choice. Erlang (and by association Elixir) has Dialyzer http://www.erlang.org/doc/man/dialyzer.html http://www.erlang.org/doc/man/dialyzer.html which is a static code analyzer. Static code analysis has been endorsed by John Carmack (and of course, all the 3 year old links to the blog post are now dead, sigh) For anyone (like me) who is hooked on Elixir, https://github.com/jeremyjh/dialyxir https://github.com/jeremyjh/dialyxir provides a bit of easier Dialyzer management Elixir (and I suppose Erlang) has this concept of "specs" which you don't have to use, but which help code analysis if you do... http://elixir-lang.org/docs/v1.0/elixir/Kernel.Typespec.html http://elixir-lang.org/docs/v1.0/elixir/Kernel.Typespec.html

kazinator 11y ago

This is misleadingly titled; it should be called: Rules, Mostly Pertaining to C, For Writing Safety-Critical Code (with Rule 0 Removed) (Rule 0 is: Don't use C for safety-critical code!) (And you know there was a Rule [0], obviously!)