7 ms·

I know that, at different times, NASA has used Forth[1] and Lisp[2] in some of their space applications. Both of these languages offer REPLs that generally acce

by fouric 5y ago

I know that, at different times, NASA has used Forth[1] and Lisp[2] in some of their space applications. Both of these languages offer REPLs that generally accelerate the debugging process, and while your "average" Lisp might be unsuitable for hard real-time applications (due to the presence of a garbage collector, usually without the hard real-time constraints that you can get out of garbage collectors with extreme effort), I wonder if they have some equivalently interactive system on-board the Hubble. > Most of Hubble's components have redundant back-ups, so once scientists figure out the specific component that's causing the computer problem, they can remotely switch over to its back-up part. Wait, then why don't they just switch over each component in turn? The "divide and conquer" debugging strategy. [1] https://www.forth.com/resources/space-applications/ https://www.forth.com/resources/space-applications/ [2] https://flownet.com/gat/jpl-lisp.html https://flownet.com/gat/jpl-lisp.html

dylan604 5y ago
What if you backup is failing in the exact same way?
- Koshkin 5y ago
  That would probably mean that the whole thing just isn't there anymore.
  - dylan604 5y ago
    Or much more likely the same component was used as a back up, and is failing in a similar fashion. It's obvious the thing is still there.
beerandt 5y ago
>"The rule of thumb is when something is working you don't change it," Hertz said. "We'd like to change as few things as possible when we bring Hubble back into service."
- AnIdiotOnTheNet 5y ago
  A lesson not taught to any modern software developer. Instead they change things all the time for no real reason other than that they want to change things.
  - 35fbe7d3d5b9 5y ago
    One of the best senior engineers I worked with taught me how to run an outage. The most important thing? Stop what you are doing, take charge, and get everyone else to stop what they are doing. The best case scenario of a bunch of engineers flailing about on a bridge turning knobs is that you luck into a fix but don't know how you got there. But you're more likely to make things worse.
    - londons_explore 5y ago
      In my experience, the best strategy depends a lot on the severity of the outage. If all the alarms are going off because of a loss of redundancy, then currently there is no outage. The correct move should be carefully considered, and maybe tested in the sandbox environment. If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy.
      - ComputerGuru 5y ago
        > If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy. Maybe. Once an outage has happened, an additional 30 minutes or hour of outage, depending 100% on the service in question, might be a bearable cost if it means preventing a domino effect of future issues caused by measures taken to restore the outage in a hurry.
      - CGamesPlay 5y ago
        > If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy. Almost assuredly not. If a system hits a 100% outage, there are about to be a series of cascading failures by dependent systems. If you don't even understand which system is the root cause, all you're doing is testing a bunch of never-before-tried combinations in production and hoping something works.
etskinner 5y ago
> Wait, then why don't they just switch over each component in turn? The "divide and conquer" debugging strategy. My guess would be that they want to try that method only if this debugging doesn't work. Imagine that there's an electrical issue in item 1 that fries item 2. If you switch over to item 2b, then you fry item 2b too! This is exactly what happened with the Soviet Salyut 7 station. They tripped an over-current protection, didn't fix the root issue, and remotely turned the circuit back on. A series of electrical shorts then rendered the entire station without power, resulting in the need for one of the most daring station rescue stories of all time: https://arstechnica.com/science/2014/09/the-little-known-soviet-mission-to-rescue-a-dead-space-station/ https://arstechnica.com/science/2014/09/the-little-known-sov...
- voldacar 5y ago
  Wow, that's an amazing story. Thanks for posting, i had no idea something like that ever took place
- dmckeon 5y ago
  > Mission controllers, very tired now that the end of their 24-hour shift was approaching Are shifts this long still common practice in US or RU space programs?
baryphonic 5y ago
> while your "average" Lisp might be unsuitable for hard real-time applications (due to the presence of a garbage collector, usually without the hard real-time constraints that you can get out of garbage collectors with extreme effort), I wonder if they have some equivalently interactive system on-board the while your "average" Lisp might be unsuitable for hard real-time applications (due to the presence of a garbage collector, usually without the hard real-time constraints that you can get out of garbage collectors with extreme effort), I wonder if they have some equivalently interactive system on-board the Hubble. This is fascinating to me. Do you have any pointers to information/research/projects focused on hard real-time garbage collection? A Lisp with hard real-time garbage collection (even if Herculean to implement) would be fantastic.
- mietek 5y ago
  How about a Lisp without the need for garbage collection at all? http://web.archive.org/web/20020331165324/http://home.pipeline.com/~hbaker1/LinearLisp.html http://web.archive.org/web/20020331165324/http://home.pipeli...
  - baryphonic 5y ago
    This is also fascinating. Thank you! :)
  - retrac 5y ago
    Ah, Rust's grandparent!
- guenthert 5y ago
  Not exactly real time (as in proven bounded response time), but a noteworthy effort: https://franz.com/services/conferences_seminars/jlugm00/conference/Talk14_takeuchi.pdf https://franz.com/services/conferences_seminars/jlugm00/conf...
- astrange 5y ago
  As long as there aren't cycles of course you can do deterministic GC, it's just reference counting. It also helps if the program is single-threaded since otherwise any memory allocation/freeing can be unpredictable (since it probably locks.)
rtkwe 5y ago
> Wait, then why don't they just switch over each component in turn? The "divide and conquer" debugging strategy. It's better to understand the problem than to just start changing stuff hoping you find the right thing even systematically. There's not a huge rush to fix this since it's the payload computer and the telescope is still being maintained by other systems. A lot bad could happen, if the switching system is flakey you could maybe get stuck in a bad system, or if there's a number of faults you might damage one of the backups. Without the shuttle there's not a plan to service it any more so why take the risk rushing through to the most simplistic debugging method?
edgeform 5y ago
> Wait, then why don't they just switch over each component in turn? The "divide and conquer" debugging strategy. Let's say the CPU is the actual issue, but the problem manifests itself in the memory module. You swap over to the backup memory module, and suddenly the problem vanishes! Two months later, the problem manifests again. Identical presentation. This time, there is no backup to switch over to test. You fly a Very Expensive Mission to the telescope only to find out the CPU was the issue, and if you had figured that out originally you wouldn't be up here with four memory modules.
- nucleardog 5y ago
  Let's say the memory is the actual issue, but it's manifested itself by data being corrupted and triggering undesired behaviour. Unfortunately, the corrupted state has already been written back to persistent storage. So you swap in the backup storage module and all your problems go away. Until it happens again and corrupts _that_ too.
skylanh 5y ago
Another comment keyed onto a concrete example of why not, so I'll go with a presumptive reason: Some of those elements will be part of the major service windows, and have expected operational and standby lifetimes. So if a component with two elements has a service window of 10 years, and each element contributes to meeting that service window, then you've bumped your major service window from 10 years to a significant factor less than that. e.g. the expected use profile might be: use element 1 for 6 years or 60% of service, switch to element 2 for 4 years, replace both during 10 year maintenance window. Interrupting that by bringing element 2 up reduces that window and contingency plans if the service window cannot be met. I don't know, and I'm just talking out my you-know-what.
rurban 5y ago
Forth yes, lisp not. Lisp was only used on ground to simulate the rover. Also a repl in space only makes sense in earth orbit, but not farther away, with 8-20min waiting time for a packet roundtrip to Mars. Those machines really need proper and faster decision making (AI, think lots of `if` statements and proper modeling) on board, eg to perform landing or docking maneuvers. Or to detect and workaround radiation damage in its own circuits.
- dchristian 5y ago
  You might check on how New Horizons was implemented. They initially wanted to fly LISP, but (I think) they eventually used the LISP to generate C++. This was to allow dynamic, model based re-planning when the craft was in the outer reaches of the solar system.

by fouric 5y ago

dylan604 5y ago

What if you backup is failing in the exact same way?

Koshkin 5y ago
That would probably mean that the whole thing just isn't there anymore.
- dylan604 5y ago
  Or much more likely the same component was used as a back up, and is failing in a similar fashion. It's obvious the thing is still there.

beerandt 5y ago

>"The rule of thumb is when something is working you don't change it," Hertz said. "We'd like to change as few things as possible when we bring Hubble back into service."

AnIdiotOnTheNet 5y ago
A lesson not taught to any modern software developer. Instead they change things all the time for no real reason other than that they want to change things.
- 35fbe7d3d5b9 5y ago
  One of the best senior engineers I worked with taught me how to run an outage. The most important thing? Stop what you are doing, take charge, and get everyone else to stop what they are doing. The best case scenario of a bunch of engineers flailing about on a bridge turning knobs is that you luck into a fix but don't know how you got there. But you're more likely to make things worse.
  - londons_explore 5y ago
    In my experience, the best strategy depends a lot on the severity of the outage. If all the alarms are going off because of a loss of redundancy, then currently there is no outage. The correct move should be carefully considered, and maybe tested in the sandbox environment. If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy.
    - ComputerGuru 5y ago
      > If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy. Maybe. Once an outage has happened, an additional 30 minutes or hour of outage, depending 100% on the service in question, might be a bearable cost if it means preventing a domino effect of future issues caused by measures taken to restore the outage in a hurry.
    - CGamesPlay 5y ago
      > If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy. Almost assuredly not. If a system hits a 100% outage, there are about to be a series of cascading failures by dependent systems. If you don't even understand which system is the root cause, all you're doing is testing a bunch of never-before-tried combinations in production and hoping something works.

etskinner 5y ago

> Wait, then why don't they just switch over each component in turn? The "divide and conquer" debugging strategy. My guess would be that they want to try that method only if this debugging doesn't work. Imagine that there's an electrical issue in item 1 that fries item 2. If you switch over to item 2b, then you fry item 2b too! This is exactly what happened with the Soviet Salyut 7 station. They tripped an over-current protection, didn't fix the root issue, and remotely turned the circuit back on. A series of electrical shorts then rendered the entire station without power, resulting in the need for one of the most daring station rescue stories of all time: https://arstechnica.com/science/2014/09/the-little-known-soviet-mission-to-rescue-a-dead-space-station/ https://arstechnica.com/science/2014/09/the-little-known-sov...

voldacar 5y ago
Wow, that's an amazing story. Thanks for posting, i had no idea something like that ever took place
dmckeon 5y ago
> Mission controllers, very tired now that the end of their 24-hour shift was approaching Are shifts this long still common practice in US or RU space programs?

baryphonic 5y ago

> while your "average" Lisp might be unsuitable for hard real-time applications (due to the presence of a garbage collector, usually without the hard real-time constraints that you can get out of garbage collectors with extreme effort), I wonder if they have some equivalently interactive system on-board the while your "average" Lisp might be unsuitable for hard real-time applications (due to the presence of a garbage collector, usually without the hard real-time constraints that you can get out of garbage collectors with extreme effort), I wonder if they have some equivalently interactive system on-board the Hubble. This is fascinating to me. Do you have any pointers to information/research/projects focused on hard real-time garbage collection? A Lisp with hard real-time garbage collection (even if Herculean to implement) would be fantastic.

mietek 5y ago
How about a Lisp without the need for garbage collection at all? http://web.archive.org/web/20020331165324/http://home.pipeline.com/~hbaker1/LinearLisp.html http://web.archive.org/web/20020331165324/http://home.pipeli...
- baryphonic 5y ago
  This is also fascinating. Thank you! :)
- retrac 5y ago
  Ah, Rust's grandparent!
guenthert 5y ago
Not exactly real time (as in proven bounded response time), but a noteworthy effort: https://franz.com/services/conferences_seminars/jlugm00/conference/Talk14_takeuchi.pdf https://franz.com/services/conferences_seminars/jlugm00/conf...
astrange 5y ago
As long as there aren't cycles of course you can do deterministic GC, it's just reference counting. It also helps if the program is single-threaded since otherwise any memory allocation/freeing can be unpredictable (since it probably locks.)

rtkwe 5y ago

> Wait, then why don't they just switch over each component in turn? The "divide and conquer" debugging strategy. It's better to understand the problem than to just start changing stuff hoping you find the right thing even systematically. There's not a huge rush to fix this since it's the payload computer and the telescope is still being maintained by other systems. A lot bad could happen, if the switching system is flakey you could maybe get stuck in a bad system, or if there's a number of faults you might damage one of the backups. Without the shuttle there's not a plan to service it any more so why take the risk rushing through to the most simplistic debugging method?

edgeform 5y ago

> Wait, then why don't they just switch over each component in turn? The "divide and conquer" debugging strategy. Let's say the CPU is the actual issue, but the problem manifests itself in the memory module. You swap over to the backup memory module, and suddenly the problem vanishes! Two months later, the problem manifests again. Identical presentation. This time, there is no backup to switch over to test. You fly a Very Expensive Mission to the telescope only to find out the CPU was the issue, and if you had figured that out originally you wouldn't be up here with four memory modules.

nucleardog 5y ago
Let's say the memory is the actual issue, but it's manifested itself by data being corrupted and triggering undesired behaviour. Unfortunately, the corrupted state has already been written back to persistent storage. So you swap in the backup storage module and all your problems go away. Until it happens again and corrupts _that_ too.

skylanh 5y ago

Another comment keyed onto a concrete example of why not, so I'll go with a presumptive reason: Some of those elements will be part of the major service windows, and have expected operational and standby lifetimes. So if a component with two elements has a service window of 10 years, and each element contributes to meeting that service window, then you've bumped your major service window from 10 years to a significant factor less than that. e.g. the expected use profile might be: use element 1 for 6 years or 60% of service, switch to element 2 for 4 years, replace both during 10 year maintenance window. Interrupting that by bringing element 2 up reduces that window and contingency plans if the service window cannot be met. I don't know, and I'm just talking out my you-know-what.

rurban 5y ago

Forth yes, lisp not. Lisp was only used on ground to simulate the rover. Also a repl in space only makes sense in earth orbit, but not farther away, with 8-20min waiting time for a packet roundtrip to Mars. Those machines really need proper and faster decision making (AI, think lots of `if` statements and proper modeling) on board, eg to perform landing or docking maneuvers. Or to detect and workaround radiation damage in its own circuits.

dchristian 5y ago
You might check on how New Horizons was implemented. They initially wanted to fly LISP, but (I think) they eventually used the LISP to generate C++. This was to allow dynamic, model based re-planning when the craft was in the outer reaches of the solar system.