4 ms·

I'm sure Ada makes a reliable program, but wouldn't it also be good to have an Erlang/Elixir style "self-healing" system with supervisors that can stop and rest

by olah_1 5y ago

I'm sure Ada makes a reliable program, but wouldn't it also be good to have an Erlang/Elixir style "self-healing" system with supervisors that can stop and restart each other?

Jtsummers 5y ago
That actually is done with a lot of these kinds of systems. In many of the ones I've worked on you have software or hardware "watchdog timers" that have to be reset, or a process will be terminated and restarted (how will vary a bit). Hardware watchdog timers usually trigger something like a reboot, causing the program to be reloaded and restarted. That's about the simplest you can make a supervisor, but you can have more complex ones monitoring the actual internals to respond should the state become corrupted in some fashion as well. https://en.wikipedia.org/wiki/Watchdog_timer https://en.wikipedia.org/wiki/Watchdog_timer
- trothamel 5y ago
  The programs that ran on the Apollo Guidance Computer were written this way. If there was a sufficiently bad error, the AGC would restart execution. The programs took advantage of the memory in use at the time being non-volatile, and not reset to zero when the computer restarted, so the program could figure out what the CSM or LM was doing, and continue it. This actually happened on Apollo 11. The 1201 and 1202 program alarms were the computer overloading, and running out of space for more processes. When this happened, the computer would reset, and navigation would pick up again.
- touisteur 5y ago
  To add to this, usually most of the systems in avionics or support systems on the ground are dual-hardware (when not more). There's lots of work done to ensure there's no single point of failure and hardware (and more and more, software) failures are handled safely. Regulators add every 5-10y some pressure on time-to-recovery and acceptable data of service loss (counted in milliseconds even on ground systems). Erlang didn't invent much there, and restarting/rebooting isn't always possible (try doing that in less than 10ms). Sometimes, the application itself needs to handle problems. This puts lots of 'classic IT' (interface bonding, fail overs, etc.) stuff to the bin very quick.
SkyMarshal 5y ago
Ada/Spark can do that too, but you also want things like memory safety, hard real-time and/or constant-time capability, and provable correctness. Ada also enables those capabilities.
Verdex 5y ago
I'm interested in hearing more. I've been looking into erlang on and off for nearly 14 years now. I think the language is really neat. But the self-healing supervisor tree thing is something that I've yet to get my head wrapped around. Like ... if you have some code that's like "X / 0" then it seems like it's not going to matter how many times you restart the process.
- macintux 5y ago
  The general strategy is to test the system to the point where any bug like that is an edge case, some weird input you didn’t expect. So every time you get that input you’ll still crash, but other transactions will continue normally. Hopefully that input is either due to a transitory glitch like a bit flipped in RAM, or a user who isn’t bored enough to keep submitting it, but either way the process isolation means all your other jobs can continue.
MaxBarraclough 5y ago
I'm not sure that mindset fits for this kind of problem. It's not a web server handling lots of independent requests, it's a real-time critical system that is required to always transition to the correct state within the deadline. I'm no expert on this stuff though.
- deleted 5y ago
  [deleted]

by olah_1 5y ago

I'm sure Ada makes a reliable program, but wouldn't it also be good to have an Erlang/Elixir style "self-healing" system with supervisors that can stop and restart each other?

Jtsummers 5y ago

That actually is done with a lot of these kinds of systems. In many of the ones I've worked on you have software or hardware "watchdog timers" that have to be reset, or a process will be terminated and restarted (how will vary a bit). Hardware watchdog timers usually trigger something like a reboot, causing the program to be reloaded and restarted. That's about the simplest you can make a supervisor, but you can have more complex ones monitoring the actual internals to respond should the state become corrupted in some fashion as well. https://en.wikipedia.org/wiki/Watchdog_timer https://en.wikipedia.org/wiki/Watchdog_timer

trothamel 5y ago
The programs that ran on the Apollo Guidance Computer were written this way. If there was a sufficiently bad error, the AGC would restart execution. The programs took advantage of the memory in use at the time being non-volatile, and not reset to zero when the computer restarted, so the program could figure out what the CSM or LM was doing, and continue it. This actually happened on Apollo 11. The 1201 and 1202 program alarms were the computer overloading, and running out of space for more processes. When this happened, the computer would reset, and navigation would pick up again.
touisteur 5y ago
To add to this, usually most of the systems in avionics or support systems on the ground are dual-hardware (when not more). There's lots of work done to ensure there's no single point of failure and hardware (and more and more, software) failures are handled safely. Regulators add every 5-10y some pressure on time-to-recovery and acceptable data of service loss (counted in milliseconds even on ground systems). Erlang didn't invent much there, and restarting/rebooting isn't always possible (try doing that in less than 10ms). Sometimes, the application itself needs to handle problems. This puts lots of 'classic IT' (interface bonding, fail overs, etc.) stuff to the bin very quick.

SkyMarshal 5y ago

Ada/Spark can do that too, but you also want things like memory safety, hard real-time and/or constant-time capability, and provable correctness. Ada also enables those capabilities.

Verdex 5y ago

I'm interested in hearing more. I've been looking into erlang on and off for nearly 14 years now. I think the language is really neat. But the self-healing supervisor tree thing is something that I've yet to get my head wrapped around. Like ... if you have some code that's like "X / 0" then it seems like it's not going to matter how many times you restart the process.

macintux 5y ago
The general strategy is to test the system to the point where any bug like that is an edge case, some weird input you didn’t expect. So every time you get that input you’ll still crash, but other transactions will continue normally. Hopefully that input is either due to a transitory glitch like a bit flipped in RAM, or a user who isn’t bored enough to keep submitting it, but either way the process isolation means all your other jobs can continue.

MaxBarraclough 5y ago

I'm not sure that mindset fits for this kind of problem. It's not a web server handling lots of independent requests, it's a real-time critical system that is required to always transition to the correct state within the deadline. I'm no expert on this stuff though.

deleted 5y ago
[deleted]