9 ms·

Erlang programmer’s view on Curiosity Rover software

196 points by deno 14y ago

rubyrescue 14y ago
Great article. The only thing he left out is the parallel to Erlang Supervisor Trees, which give the ability to restart parts of the system that have failed in some way without affecting the overall system.
jeremiep 14y ago
Great article! I'd like to add that the D programming language also offers a lot of features to create robust code with multiple paradigms, although the syntax is heavily C oriented rather than functional. 'immutable' and 'shared' are added to the known C 'const' qualifier for data that will never change (contrary to not changing in the declaring scope only) data which is shared across threads, everything else is encouraged to use MPI using the std.concurrency module. Pure functional code can be enforced by the compiler by using the 'pure' qualifier. There is even compile time function evaluation if called with constant arguments, which is awesome when used with it's type-safe generic and meta-programming. There's unit tests, contracts, invariants and documentation support right in the language. Plus the compiler does profiling and code coverage. I'd be curious to test D against Erlang for such a system. (Not saying Erlang shouldn't be used, it's the next language on my to-learn list, just that the switch to functional might be too radical for most developers used to imperative and OO and D provides the best of both worlds.)
- misnome 14y ago
  I've been interested in D for a while, for these reasons and more - it's features look nice, but it never seems to have gotten much popularity/mindshare. Could you hazard a guess why?
  - pjmlp 14y ago
    In the early days there were some issues in the community which lead to the Phobos/Tango divide in the standard library for D1. This is now past history as the community has joined around D2, just known as D, and strives to reach compliancy with the "The D Programming Language" book written by Andrei Alexandrescu. D2 development is made in the open with source code available in GitHub. Besides the reference compiler, dmd owned by Digital Mars, there are also LDC and GDC compilers available. Currently it seems that GDC might be integrated into GCC as of 4.8 release. Right now it seems more people are picking up D, mainly for game development projects.
    - CJefferson 14y ago
      Indeed, the D1 splits, and and also questions being answered with "that will be in D2". There is now a problem that there is not yet a consensus about how D2 compares to C++11.
DanielBMarkham 14y ago
Message-passing better than STM? Wonder why?
- matthavener 14y ago
  I think two reasons: 1) VxWorks directly supports message passing (http://www.vxdev.com/docs/vx55man/vxworks/ref/msgQLib.html http://www.vxdev.com/docs/vx55man/vxworks/ref/msgQLib.html). 2) They seem to prefer simple, obvious, "less magic" interfaces. STM is nice for its "magic", but message creates very well defined, documented interfaces between code.
  - deno 14y ago
    > STM is nice for its "magic", but message creates very well defined, documented interfaces between code. The author’s previous post has a good overview of how message passing contributes to that as well. http://jlouisramblings.blogspot.com/2012/06/protocols-in-kingdom-of-distribution.html http://jlouisramblings.blogspot.com/2012/06/protocols-in-kin...
- jlouis 14y ago
  Different. If you squint your eyes enough, the Erlang mnesia application is an implementation of (distributed) STM. STM and MP are not a priori against each other. You can have both and sometimes one is better than the other for a certain kinds of problems.
matthavener 14y ago
The biggest difference to Erlang is VxWork's inability to isolate task faults or runaway high priority process. (Tasks are analogous to Processes in Erlang). VxWorks 6.0 supports isolation to some degree, but it was released in '04, after the design work on the rover started. Without total isolation, a lot of the supervisor benefits of VxWorks goes away.
- vbtemp 14y ago
  Hm.. What do you mean by isolating task faults? I think a lot of that depends on the underlying hardware, right (e.g., if the board has an MMU)? I know you can insert a taskSwitchHook (I think it's called) that could be able to detect and kill runaway high-priority processes. Edit: in response to the reply, I suppose I should have mean tasks instead of "processes" (which in VxWorks would be the RTP)
  - noselasd 14y ago
    VxWorks has no processes[1]. It has tasks. Basically, you write kernel code, there's no user mode. [1] As someone mentioned, VxVorks 6 did introduce processes and "usermode", called RTP. As with most features of VxWorks, you compile that into your image if you want the feature. But there's a lot of inertia, and much of the VxWorks stuff I see doesn't use RTP yet.
- deno 14y ago
  Erlang could use a bit more isolation as well. For example limiting the mailbox size and limiting the memory available to a single process. Memory limits would be great to give you some ease of mind when reusing libraries for processing input. For example processing XML, even using SAX parser, still leaves you open for attacks like huge tag values or just something more real world, huge href="data:" attributes. Obviously the right way is to have a limit in the parser (which xmerl doesn’t have, unfortunately), in this case, but it’d be nice to have some safety net from the VM as well.
ricardobeat 14y ago
So a Mars Rover is much closer to a browser/backbone/node.js app than I could ever imagine. The basic structure is surprisingly similar to javascript apps these days: isolated modules, message passing/event loop, fault tolerance.
- taybin 14y ago
  If a function call in a node.js blocks, doesn't the app hang while waiting for it? A single process with event callbacks is almost the opposite of what is described here.
  - jeremiep 14y ago
    In node.js operation that would normally block are performed in a worker thread. For example to read a file you specify a filename and a callback function that will be called when the file is in memory. After the call the execution of the current thread will eventually return to the event loop and other events will be processed until the file is read, such as requests or other callbacks. Once the file is read, a call to the specified callback is queued on the event loop and then executed, resuming your code.
  - ricardobeat 14y ago
    My comment was specifically about module isolation and message-passing, the design patterns, not language implementation, preemptive vs cooperative or whatever.
    - taybin 14y ago
      I hate to harp on this, since you were downvoted to oblivion, but node.js doesn't have module isolation, as a crash in one function will stop the entire program. And it doesn't have message-passing since it doesn't have multiple processes. There's nothing to pass messages to. The only design pattern it shares is splitting code into subsets, and every language (that matters) has that.
      - 14y ago
deleted 14y ago
[deleted]
- deno 14y ago
  The figure is just mentioned to say that the code base is quite complex. And what are you talking about? It’s not a rant on anything. Certainly not the 2.5 MLOC.
- zaptheimpaler 14y ago
  Ah, yet another thoughtless comment based purely on reading the first few lines.
vbtemp 14y ago
The motivation for writing the software in C is this: Code Reuse. NASA and it's associated labs have produced some rock solid software in C. In space missions commonly the RAD750 is used (with it's non-hardened version, the MCP), along with the Leon family of processors. Test beds and other ground hardware are often little-endian Intel processors. VxWorks is commonly used on many missions and ground systems, but so is QNX, Linux, RTEMS, etc... The only common thing the diverse set of hardware, operating systems, and compiler tool chains all support is ANSI C. This means that nifty languages like Erlang or whatever - though there may be a solid case for using them - is not practical in this circumstance. I know some clever folks in the business have done interesting work on ML-to-C compilers, but it's still in the early R&D phase at this point - the compiler itself would have to be thoroughly vetted.
- jensnockert 14y ago
  I didn't read it as arguing against C, just noting that there seems to be a lot of commonality between the ways the code in the Mars rovers are designed, and the way that robust Erlang applications are typically designed.
  - jlouis 14y ago
    Precisely. One thing very must against using Erlang for this problem is that you need hard real-time behaviour. Erlang does not provide that. The other point, you need static allocation almost everywhere, is also detrimental to using Erlang for the Rovers. That leaves you with very few languages you can use, and C is a good predictable one for the problem. Its tool support is also quite good with static verification etc. And it is a good target for compilation. As someone else notes, most of those 2.5 Megalines are auto-generated.
- angrycoder 14y ago
  reading the article is awesome.
- _delirium 14y ago
  Since much of those 2.5-MLOC are auto-generated, wouldn't it be accurate to say that they did use a something-to-C compiler, only instead of, say, ML-to-C, it was their own in-house something-to-C compiler? And that 'something' seems to have borrowed a number of features from Erlang.
thepumpkin1979 14y ago
deno, I was wondering, if so similar to erlang, why not use erlang instead C? What is the major drawback, footprint?
- sausagefeet 14y ago
  Hard realtime and control over memory allocation. Also, Erlang is not necessarily great when it comes to static analysis.
- deno 14y ago
  I’m not the author, so let me quote: > It turns out that many of the traits of Erlang systems overlap with that of the Rovers. But I don't think this is a coincidence. The software has certain different properties — the rovers are hard realtime whereas the erlang systems are soft realtime. > The things that do not overlap has to do with the need of having soft realtime vs hard realtime. In Erlang we can yield service. It is bad, but we can do it. On a rover it can be disastrous. Especially in the flight control software. Fire a rocket too late and you are in trouble.
- malkia 14y ago
  The OTP virtual machine takes a lot of memory. It's an interpretter, which means much slower execution, as article and others above pointed - The Erlang VM is soft-realtime, it can't guarantee that something would finish in certain amount of micro or milliseconds, or if it does guarantee - it's too much for what they need (just guessing here). But the concepts are very similar - message passing being the way to communicate between modules, rather than shared memory ways. This brings another topic - the Linux vs Minix debate :) - I guess there are right things to be done for the right time, and right target. It's just getting all these things right is the hardest.
  - deno 14y ago
    > It's an interpretter, which means much slower execution Only BEAM is an interpreter, there are HiPE and ErlLLVM backends as well. You can also write NIFs — functions in C that can be executed within VM.
- 14y ago
sausagefeet 14y ago
Does anyone have any knowledge of why Ada isn't used over C? Specifically, it seems like Ada gives you a lot better tools when it comes to numerical overflows/underflows. Also, what compiler does NASA use? Something like CompCert? What kind of compiler flags? Do they run it through an optimizer at all?
- vbtemp 14y ago
  See my post below - to reuse code cross platform. There's a diverse set of compiler toolchains, operating systems, architectures. Only ANSI C is supported by all of them. The compilers are specific to the target OS and hardware, and flags are unsurprisingly the strictest possible for C89.
  - ibotty 14y ago
    i'd think that you can run ada generated code pretty much everywhere. even on obscure hardware that works in space.
- _sh 14y ago
  As to what compiler NASA uses, their C Coding Standard [1] nominates a set of flags for GCC (page 8). http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf
  - sausagefeet 14y ago
    Interesting, I'm curious if they have looked into CompCert, and if so what they think of it. Maybe it doesn't target the architecture they want. There is also vellvm which seems like something a space mission would care about. Although, I've never heard of a gcc compiler bug being the cause of a NASA mission failure so perhaps gcc is Good Enough?
    - sanxiyn 14y ago
      CompCert does support PowerPC which RAD750 implements. On the other hand, compiler bugs are probably not the weakest link.
1gor 14y ago
Any _robust_ C program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Erlang... http://c2.com/cgi/wiki?GreenspunsTenthRuleOfProgramming http://c2.com/cgi/wiki?GreenspunsTenthRuleOfProgramming
- gruseom 14y ago
  But in this case that's almost completely wrong. For example, "bug-ridden"? Outfits like NASA use classical techniques (code inspection etc.) to ensure that their software has exceedingly low error rates. This has been well studied. Such an approach works, it's just too expensive for most commercial projects. As for "slow", how likely is that? On another note, it's pretty cool that the first three names credited in the JPL coding standard document (which is linked to at the bottom of the OP and is surprisingly well written) are Brian Kernighan, Dennis Ritchie, and Doug McIlroy.
  - makmanalp 14y ago
    > As for "slow", how likely is that? Have you seen the rover move? :) http://www.youtube.com/watch?v=484R2TQGLAQ http://www.youtube.com/watch?v=484R2TQGLAQ Jokes aside, there isn't much of a speed requirement here, really. It all runs fine on a radiation-hardened 200mhz processor. Nothing on mars is running away.
  - GuiA 14y ago
    >But in this case that's almost completely wrong. On the second line of linked article: "This is a humorous observation" (emphasis mine :) )
    - gruseom 14y ago
      Humor doesn't make it applicable. Greenspun's line had a specific meaning. It doesn't stick to this surface.
  - 4ad 14y ago
    > On another note, it's pretty cool that the first three names credited in the JPL coding standard document are Brian Kernighan, Dennis Ritchie, and Doug McIlroy. It's not surprising at all. It's the work of Gerard Holzmann[1], who worked at Bell Labs at the Computing Sciences Research Center with all the other Unix folks where he developed the Spin[2] model checker. (As a sidenote, Spin was used to check the sanity of the Plan 9 kernel). [1] http://spinroot.com/gerard/ http://spinroot.com/gerard/ [2] http://spinroot.com/spin/whatispin.html http://spinroot.com/spin/whatispin.html
pgeorgi 14y ago
"We know that most of the code is written in C and that it comprises 2.5 Megalines of code, roughly[1]. One may wonder why it is possible to write such a complex system and have it work. This is the Erlang programmers view." Contrast this with https://www.ohloh.net/p/erlang https://www.ohloh.net/p/erlang: "In a Nutshell, Erlang has had 7,332 commits made by 162 contributors representing 2,346,438 lines of code" I'm not sure if those roughly 154kloc really make a difference...
- deno 14y ago
  The 2.5 MLOC of NASA code is mostly generated. I’m not really sure what point are you trying to make…?
  - deleted 14y ago
    [deleted]
  - pgeorgi 14y ago
    From an erlang programmer's point of view, 2.5MLOC are a complex system. On the other hand, every Hello World in Erlang drags in about 2.5MLOC of liability (even if much of that is never run). And I doubt it's all autogenerated. So if anything, 2.5MLOC of generated NASA code is probably less complex than the erlang runtime.
    - deno 14y ago
      The article is not about the MLOC. And anyway, that 2.5 MLOC of Erlang includes all kind of libraries, like Mnesia, an entire database application.
- ibotty 14y ago
  you cannot compare total code size with the size of the repository.
Tloewald 14y ago
Back in the 90s there was a software engineering fad (unfair term but it was faddish at the time) called the process maturity index, and JPL was one of two software development sites that qualified for the highest rank (5) which involves continuous improvement, measuring everything, and going from rigorous spec to code via mathematical proof. This process (which Ed Jourdan neatly eviscerated when applied to business software) produces software that is as reliable as the specification and underlying hardware.
- vonmoltke 14y ago
  It may be a fad for the industry at large, but it's a requirement in US government contracting (as CMMI). It goes beyond software, too. My former employer just got their systems engineering up to CMMI Level 5[1] and was working hard on getting electrical engineering there (they are only at 3). [1] Software had been there for a few years.
davidw 14y ago
Great article and comparison, and a nice way of highlighting one of Erlang's strengths. However: I'm dubious that it's a strength many people here need. No, the article did not say anything about that, but I am. A few minutes of downtime, now and then, for a web site that's small and iterating rapidly to find a good market fit, is not the biggest problem. And while Erlang isn't bad at that, I don't think it's as fast as something like Rails to code in, and have all kinds of stuff ready to go out of the box. That said, I'd still recommend learning the language, just because it's so cool how it works under the hood, and because sooner or later, something will take its best ideas and get popular, so having an idea how that kind of thing works will still be beneficial.
- timClicks 14y ago
  As you mentioned Rails, I thought I should mentioned Chicago Boss. It's a blindingly fast Rails-inspired framework that takes many Erlangisms out of coding in Erlang: http://chicagoboss.org/ http://chicagoboss.org/
donpdonp 14y ago
"Recursion is shunned upon for instance,...message passing is the preferred way of communicating between subsystems....isolation is part of the coding guidelines... The Erlang programmer nods at the practices." Best "Smug Programmer" line ever.
iRobot 14y ago
with a reliable system ready to take over at any instance, you think it would be a good idea to run with a much higher spec "less reliable" systems during the hours where the temperature/radiation/etc is within the normal spec of the lesser system. Just a thought.
pron 14y ago
I absolutely love Erlang and think that, along with Clojure, it provides a complete ideology for developing modern software. But the article implies (and more than once) that the rover's architecture borrows from Erlang, while the opposite is true. Erlang adopted common best practices from fault-tolerant, mission-critical software, and packaged them in a language and runtime that make deviating from those principles difficult. The rover's software shows Erlang's roots, not its legacy.
- jlouis 14y ago
  It wasn't the intention to imply that one borrowed from the other. The concept of isolation and virtual memory protection goes back to at least the 60'es for instance. It would be unwise not to take the knowledge into account - even though we are extremely good at forgetting our past in this industry :/ The key take away is that these methods seems to work. They worked in a setting of C code on the existing rovers as well as the newest one. They worked on large telephony switches written in Erlang. My guess would be that many other mission critical systems will posses the same traits.
- itmag 14y ago
  I have been wanting to get started with Erlang for a while now. Mind telling me a bit about features I can look forward to? What makes Erlang great?
  - pron 14y ago
    Erlang is best suited for online applications (as opposed to batch processing) that require handling a high-rate of concurrent events and need to be fault-tolerant (highly available). Erlang was designed with those applications in mind, and provides just the right abstractions (and no more) for implementing them correctly and efficiently (like light-weight processes with mailboxes, complete process isolation, and supervision hierarchy). But what I like most about Erlang (and I like Clojure for the same reason) is that it is not a kitchen-sink aggregation of programming language features thought to be useful or cool (like some other languages that I won't mention), but a perfect (or near-perfect) mix of just the right features, all made to work with one another in synergy. In other words, Erlang, like Clojure, does not say, "here are the features you can program with", but rather, "this is how you should program". And the features these two languages provide are, I think, just the right ones for developing software with modern requirements (scaling and high-availability) on modern hardware (concurrency). Erlang's features, however, do tend to focus more on programming in the large, or high-level software organization (they helps making your software scalable and fault-tolerant) and less on low-level constructs (it's hard or impossible to write a super high-performance data structure in Erlang). It delegates those low-level problems to units written in other languages. Clojure is the opposite, I think. It's designed to make building correct and efficient data structures and data-processing functions easy, but provides little guidance to high-level organization of complex, large software. Both are also relatively easy to learn, which only shows their elegance (though Erlang does suffer from a somewhat antiquated syntax owing its origin to Prolog).

196 points by deno 14y ago

Great article. The only thing he left out is the parallel to Erlang Supervisor Trees, which give the ability to restart parts of the system that have failed in some way without affecting the overall system.

jeremiep 14y ago

Great article! I'd like to add that the D programming language also offers a lot of features to create robust code with multiple paradigms, although the syntax is heavily C oriented rather than functional. 'immutable' and 'shared' are added to the known C 'const' qualifier for data that will never change (contrary to not changing in the declaring scope only) data which is shared across threads, everything else is encouraged to use MPI using the std.concurrency module. Pure functional code can be enforced by the compiler by using the 'pure' qualifier. There is even compile time function evaluation if called with constant arguments, which is awesome when used with it's type-safe generic and meta-programming. There's unit tests, contracts, invariants and documentation support right in the language. Plus the compiler does profiling and code coverage. I'd be curious to test D against Erlang for such a system. (Not saying Erlang shouldn't be used, it's the next language on my to-learn list, just that the switch to functional might be too radical for most developers used to imperative and OO and D provides the best of both worlds.)

misnome 14y ago
I've been interested in D for a while, for these reasons and more - it's features look nice, but it never seems to have gotten much popularity/mindshare. Could you hazard a guess why?
- pjmlp 14y ago
  In the early days there were some issues in the community which lead to the Phobos/Tango divide in the standard library for D1. This is now past history as the community has joined around D2, just known as D, and strives to reach compliancy with the "The D Programming Language" book written by Andrei Alexandrescu. D2 development is made in the open with source code available in GitHub. Besides the reference compiler, dmd owned by Digital Mars, there are also LDC and GDC compilers available. Currently it seems that GDC might be integrated into GCC as of 4.8 release. Right now it seems more people are picking up D, mainly for game development projects.
  - CJefferson 14y ago
    Indeed, the D1 splits, and and also questions being answered with "that will be in D2". There is now a problem that there is not yet a consensus about how D2 compares to C++11.

DanielBMarkham 14y ago

Message-passing better than STM? Wonder why?

matthavener 14y ago
I think two reasons: 1) VxWorks directly supports message passing (http://www.vxdev.com/docs/vx55man/vxworks/ref/msgQLib.html http://www.vxdev.com/docs/vx55man/vxworks/ref/msgQLib.html). 2) They seem to prefer simple, obvious, "less magic" interfaces. STM is nice for its "magic", but message creates very well defined, documented interfaces between code.
- deno 14y ago
  > STM is nice for its "magic", but message creates very well defined, documented interfaces between code. The author’s previous post has a good overview of how message passing contributes to that as well. http://jlouisramblings.blogspot.com/2012/06/protocols-in-kingdom-of-distribution.html http://jlouisramblings.blogspot.com/2012/06/protocols-in-kin...
jlouis 14y ago
Different. If you squint your eyes enough, the Erlang mnesia application is an implementation of (distributed) STM. STM and MP are not a priori against each other. You can have both and sometimes one is better than the other for a certain kinds of problems.

matthavener 14y ago

The biggest difference to Erlang is VxWork's inability to isolate task faults or runaway high priority process. (Tasks are analogous to Processes in Erlang). VxWorks 6.0 supports isolation to some degree, but it was released in '04, after the design work on the rover started. Without total isolation, a lot of the supervisor benefits of VxWorks goes away.

vbtemp 14y ago
Hm.. What do you mean by isolating task faults? I think a lot of that depends on the underlying hardware, right (e.g., if the board has an MMU)? I know you can insert a taskSwitchHook (I think it's called) that could be able to detect and kill runaway high-priority processes. Edit: in response to the reply, I suppose I should have mean tasks instead of "processes" (which in VxWorks would be the RTP)
- noselasd 14y ago
  VxWorks has no processes[1]. It has tasks. Basically, you write kernel code, there's no user mode. [1] As someone mentioned, VxVorks 6 did introduce processes and "usermode", called RTP. As with most features of VxWorks, you compile that into your image if you want the feature. But there's a lot of inertia, and much of the VxWorks stuff I see doesn't use RTP yet.
deno 14y ago
Erlang could use a bit more isolation as well. For example limiting the mailbox size and limiting the memory available to a single process. Memory limits would be great to give you some ease of mind when reusing libraries for processing input. For example processing XML, even using SAX parser, still leaves you open for attacks like huge tag values or just something more real world, huge href="data:" attributes. Obviously the right way is to have a limit in the parser (which xmerl doesn’t have, unfortunately), in this case, but it’d be nice to have some safety net from the VM as well.

ricardobeat 14y ago

So a Mars Rover is much closer to a browser/backbone/node.js app than I could ever imagine. The basic structure is surprisingly similar to javascript apps these days: isolated modules, message passing/event loop, fault tolerance.

taybin 14y ago
If a function call in a node.js blocks, doesn't the app hang while waiting for it? A single process with event callbacks is almost the opposite of what is described here.
- jeremiep 14y ago
  In node.js operation that would normally block are performed in a worker thread. For example to read a file you specify a filename and a callback function that will be called when the file is in memory. After the call the execution of the current thread will eventually return to the event loop and other events will be processed until the file is read, such as requests or other callbacks. Once the file is read, a call to the specified callback is queued on the event loop and then executed, resuming your code.
- ricardobeat 14y ago
  My comment was specifically about module isolation and message-passing, the design patterns, not language implementation, preemptive vs cooperative or whatever.
  - taybin 14y ago
    I hate to harp on this, since you were downvoted to oblivion, but node.js doesn't have module isolation, as a crash in one function will stop the entire program. And it doesn't have message-passing since it doesn't have multiple processes. There's nothing to pass messages to. The only design pattern it shares is splitting code into subsets, and every language (that matters) has that.
    - 14y ago

deleted 14y ago

[deleted]

deno 14y ago
The figure is just mentioned to say that the code base is quite complex. And what are you talking about? It’s not a rant on anything. Certainly not the 2.5 MLOC.
zaptheimpaler 14y ago
Ah, yet another thoughtless comment based purely on reading the first few lines.

vbtemp 14y ago

The motivation for writing the software in C is this: Code Reuse. NASA and it's associated labs have produced some rock solid software in C. In space missions commonly the RAD750 is used (with it's non-hardened version, the MCP), along with the Leon family of processors. Test beds and other ground hardware are often little-endian Intel processors. VxWorks is commonly used on many missions and ground systems, but so is QNX, Linux, RTEMS, etc... The only common thing the diverse set of hardware, operating systems, and compiler tool chains all support is ANSI C. This means that nifty languages like Erlang or whatever - though there may be a solid case for using them - is not practical in this circumstance. I know some clever folks in the business have done interesting work on ML-to-C compilers, but it's still in the early R&D phase at this point - the compiler itself would have to be thoroughly vetted.

jensnockert 14y ago
I didn't read it as arguing against C, just noting that there seems to be a lot of commonality between the ways the code in the Mars rovers are designed, and the way that robust Erlang applications are typically designed.
- jlouis 14y ago
  Precisely. One thing very must against using Erlang for this problem is that you need hard real-time behaviour. Erlang does not provide that. The other point, you need static allocation almost everywhere, is also detrimental to using Erlang for the Rovers. That leaves you with very few languages you can use, and C is a good predictable one for the problem. Its tool support is also quite good with static verification etc. And it is a good target for compilation. As someone else notes, most of those 2.5 Megalines are auto-generated.
angrycoder 14y ago
reading the article is awesome.
_delirium 14y ago
Since much of those 2.5-MLOC are auto-generated, wouldn't it be accurate to say that they did use a something-to-C compiler, only instead of, say, ML-to-C, it was their own in-house something-to-C compiler? And that 'something' seems to have borrowed a number of features from Erlang.

thepumpkin1979 14y ago

deno, I was wondering, if so similar to erlang, why not use erlang instead C? What is the major drawback, footprint?

sausagefeet 14y ago
Hard realtime and control over memory allocation. Also, Erlang is not necessarily great when it comes to static analysis.
deno 14y ago
I’m not the author, so let me quote: > It turns out that many of the traits of Erlang systems overlap with that of the Rovers. But I don't think this is a coincidence. The software has certain different properties — the rovers are hard realtime whereas the erlang systems are soft realtime. > The things that do not overlap has to do with the need of having soft realtime vs hard realtime. In Erlang we can yield service. It is bad, but we can do it. On a rover it can be disastrous. Especially in the flight control software. Fire a rocket too late and you are in trouble.
malkia 14y ago
The OTP virtual machine takes a lot of memory. It's an interpretter, which means much slower execution, as article and others above pointed - The Erlang VM is soft-realtime, it can't guarantee that something would finish in certain amount of micro or milliseconds, or if it does guarantee - it's too much for what they need (just guessing here). But the concepts are very similar - message passing being the way to communicate between modules, rather than shared memory ways. This brings another topic - the Linux vs Minix debate :) - I guess there are right things to be done for the right time, and right target. It's just getting all these things right is the hardest.
- deno 14y ago
  > It's an interpretter, which means much slower execution Only BEAM is an interpreter, there are HiPE and ErlLLVM backends as well. You can also write NIFs — functions in C that can be executed within VM.
14y ago

sausagefeet 14y ago

Does anyone have any knowledge of why Ada isn't used over C? Specifically, it seems like Ada gives you a lot better tools when it comes to numerical overflows/underflows. Also, what compiler does NASA use? Something like CompCert? What kind of compiler flags? Do they run it through an optimizer at all?

vbtemp 14y ago
See my post below - to reuse code cross platform. There's a diverse set of compiler toolchains, operating systems, architectures. Only ANSI C is supported by all of them. The compilers are specific to the target OS and hardware, and flags are unsurprisingly the strictest possible for C89.
- ibotty 14y ago
  i'd think that you can run ada generated code pretty much everywhere. even on obscure hardware that works in space.
_sh 14y ago
As to what compiler NASA uses, their C Coding Standard [1] nominates a set of flags for GCC (page 8). http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf
- sausagefeet 14y ago
  Interesting, I'm curious if they have looked into CompCert, and if so what they think of it. Maybe it doesn't target the architecture they want. There is also vellvm which seems like something a space mission would care about. Although, I've never heard of a gcc compiler bug being the cause of a NASA mission failure so perhaps gcc is Good Enough?
  - sanxiyn 14y ago
    CompCert does support PowerPC which RAD750 implements. On the other hand, compiler bugs are probably not the weakest link.

1gor 14y ago

Any _robust_ C program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Erlang... http://c2.com/cgi/wiki?GreenspunsTenthRuleOfProgramming http://c2.com/cgi/wiki?GreenspunsTenthRuleOfProgramming

gruseom 14y ago
But in this case that's almost completely wrong. For example, "bug-ridden"? Outfits like NASA use classical techniques (code inspection etc.) to ensure that their software has exceedingly low error rates. This has been well studied. Such an approach works, it's just too expensive for most commercial projects. As for "slow", how likely is that? On another note, it's pretty cool that the first three names credited in the JPL coding standard document (which is linked to at the bottom of the OP and is surprisingly well written) are Brian Kernighan, Dennis Ritchie, and Doug McIlroy.
- makmanalp 14y ago
  > As for "slow", how likely is that? Have you seen the rover move? :) http://www.youtube.com/watch?v=484R2TQGLAQ http://www.youtube.com/watch?v=484R2TQGLAQ Jokes aside, there isn't much of a speed requirement here, really. It all runs fine on a radiation-hardened 200mhz processor. Nothing on mars is running away.
- GuiA 14y ago
  >But in this case that's almost completely wrong. On the second line of linked article: "This is a humorous observation" (emphasis mine :) )
  - gruseom 14y ago
    Humor doesn't make it applicable. Greenspun's line had a specific meaning. It doesn't stick to this surface.
- 4ad 14y ago
  > On another note, it's pretty cool that the first three names credited in the JPL coding standard document are Brian Kernighan, Dennis Ritchie, and Doug McIlroy. It's not surprising at all. It's the work of Gerard Holzmann[1], who worked at Bell Labs at the Computing Sciences Research Center with all the other Unix folks where he developed the Spin[2] model checker. (As a sidenote, Spin was used to check the sanity of the Plan 9 kernel). [1] http://spinroot.com/gerard/ http://spinroot.com/gerard/ [2] http://spinroot.com/spin/whatispin.html http://spinroot.com/spin/whatispin.html

pgeorgi 14y ago

"We know that most of the code is written in C and that it comprises 2.5 Megalines of code, roughly[1]. One may wonder why it is possible to write such a complex system and have it work. This is the Erlang programmers view." Contrast this with https://www.ohloh.net/p/erlang https://www.ohloh.net/p/erlang: "In a Nutshell, Erlang has had 7,332 commits made by 162 contributors representing 2,346,438 lines of code" I'm not sure if those roughly 154kloc really make a difference...

deno 14y ago
The 2.5 MLOC of NASA code is mostly generated. I’m not really sure what point are you trying to make…?
- deleted 14y ago
  [deleted]
- pgeorgi 14y ago
  From an erlang programmer's point of view, 2.5MLOC are a complex system. On the other hand, every Hello World in Erlang drags in about 2.5MLOC of liability (even if much of that is never run). And I doubt it's all autogenerated. So if anything, 2.5MLOC of generated NASA code is probably less complex than the erlang runtime.
  - deno 14y ago
    The article is not about the MLOC. And anyway, that 2.5 MLOC of Erlang includes all kind of libraries, like Mnesia, an entire database application.
ibotty 14y ago
you cannot compare total code size with the size of the repository.

Tloewald 14y ago

Back in the 90s there was a software engineering fad (unfair term but it was faddish at the time) called the process maturity index, and JPL was one of two software development sites that qualified for the highest rank (5) which involves continuous improvement, measuring everything, and going from rigorous spec to code via mathematical proof. This process (which Ed Jourdan neatly eviscerated when applied to business software) produces software that is as reliable as the specification and underlying hardware.

vonmoltke 14y ago
It may be a fad for the industry at large, but it's a requirement in US government contracting (as CMMI). It goes beyond software, too. My former employer just got their systems engineering up to CMMI Level 5[1] and was working hard on getting electrical engineering there (they are only at 3). [1] Software had been there for a few years.

davidw 14y ago

Great article and comparison, and a nice way of highlighting one of Erlang's strengths. However: I'm dubious that it's a strength many people here need. No, the article did not say anything about that, but I am. A few minutes of downtime, now and then, for a web site that's small and iterating rapidly to find a good market fit, is not the biggest problem. And while Erlang isn't bad at that, I don't think it's as fast as something like Rails to code in, and have all kinds of stuff ready to go out of the box. That said, I'd still recommend learning the language, just because it's so cool how it works under the hood, and because sooner or later, something will take its best ideas and get popular, so having an idea how that kind of thing works will still be beneficial.

timClicks 14y ago
As you mentioned Rails, I thought I should mentioned Chicago Boss. It's a blindingly fast Rails-inspired framework that takes many Erlangisms out of coding in Erlang: http://chicagoboss.org/ http://chicagoboss.org/

donpdonp 14y ago

"Recursion is shunned upon for instance,...message passing is the preferred way of communicating between subsystems....isolation is part of the coding guidelines... The Erlang programmer nods at the practices." Best "Smug Programmer" line ever.

iRobot 14y ago

with a reliable system ready to take over at any instance, you think it would be a good idea to run with a much higher spec "less reliable" systems during the hours where the temperature/radiation/etc is within the normal spec of the lesser system. Just a thought.

pron 14y ago

I absolutely love Erlang and think that, along with Clojure, it provides a complete ideology for developing modern software. But the article implies (and more than once) that the rover's architecture borrows from Erlang, while the opposite is true. Erlang adopted common best practices from fault-tolerant, mission-critical software, and packaged them in a language and runtime that make deviating from those principles difficult. The rover's software shows Erlang's roots, not its legacy.

jlouis 14y ago
It wasn't the intention to imply that one borrowed from the other. The concept of isolation and virtual memory protection goes back to at least the 60'es for instance. It would be unwise not to take the knowledge into account - even though we are extremely good at forgetting our past in this industry :/ The key take away is that these methods seems to work. They worked in a setting of C code on the existing rovers as well as the newest one. They worked on large telephony switches written in Erlang. My guess would be that many other mission critical systems will posses the same traits.
itmag 14y ago
I have been wanting to get started with Erlang for a while now. Mind telling me a bit about features I can look forward to? What makes Erlang great?
- pron 14y ago
  Erlang is best suited for online applications (as opposed to batch processing) that require handling a high-rate of concurrent events and need to be fault-tolerant (highly available). Erlang was designed with those applications in mind, and provides just the right abstractions (and no more) for implementing them correctly and efficiently (like light-weight processes with mailboxes, complete process isolation, and supervision hierarchy). But what I like most about Erlang (and I like Clojure for the same reason) is that it is not a kitchen-sink aggregation of programming language features thought to be useful or cool (like some other languages that I won't mention), but a perfect (or near-perfect) mix of just the right features, all made to work with one another in synergy. In other words, Erlang, like Clojure, does not say, "here are the features you can program with", but rather, "this is how you should program". And the features these two languages provide are, I think, just the right ones for developing software with modern requirements (scaling and high-availability) on modern hardware (concurrency). Erlang's features, however, do tend to focus more on programming in the large, or high-level software organization (they helps making your software scalable and fault-tolerant) and less on low-level constructs (it's hard or impossible to write a super high-performance data structure in Erlang). It delegates those low-level problems to units written in other languages. Clojure is the opposite, I think. It's designed to make building correct and efficient data structures and data-processing functions easy, but provides little guidance to high-level organization of complex, large software. Both are also relatively easy to learn, which only shows their elegance (though Erlang does suffer from a somewhat antiquated syntax owing its origin to Prolog).