5 ms·

As the name RAD750 indicates, that processor is designed to be radiation hardened which matters for longer missions. I doubt the Snapdragon 801 will survive as

As the name RAD750 indicates, that processor is designed to be radiation hardened which matters for longer missions. I doubt the Snapdragon 801 will survive as long or have as few errors but it also doesn't matter since Ingenuity isn't aimed for long term usage.

Robotbeat 6y ago
It’s less about longevity and more about reducing errors. Interestingly, Ingenuity has some sort of watchdog that can reset the Snapdragon in-flight fast enough to recover if there is an error.
- EarthLaunch 6y ago
  Reminds me of SpaceX or Erlang; build a process that handles errors rather than an error free process.
  - NikolaeVarius 6y ago
    I mean its space. Any computer expected to be in space and do non completely trivial things have tons of mechanisms in place to survive computer issues.
  - deleted 6y ago
    [deleted]
- ddingus 6y ago
  Presumably. That is all part of this test.
- jjoonathan 6y ago
  Reminds me of Apollo 11, where the computer kept resetting on the way down to the lunar surface but still got Neil Armstrong and Buzz Aldrin to the moon with the whole world watching.
  - kabdib 6y ago
    It wasn't resetting, it was ditching lower priority tasks that it didn't have (real) time to accomplish. Notably, the computer was still doing important work (ahem, flying the LM :-) ), and the tasks it didn't have time to do were not critical. The team had recently seen similar failures in simulation, and was able to quickly decide it was okay to proceed.
    - jhayward 6y ago
      > Notably, the computer was still doing important work (ahem, flying the LM :-) ), and the tasks it didn't have time to do were not critical. I believe this is incorrect. From an earlier HN discussion: "The 1202s were also a lot less benign than is often reported. They occurred because of the fixed two-second guidance cycle in the landing software. That is, once every two seconds, a job called the SERVICER would start. SERVICER had many tasks during the landing. In order: navigation, guidance, commanding throttle, commanding attitude, and updating displays. With an excessive load as caused by the CDU, new SERVICERs were starting before old ones could finish. Eventually there would be two many old SERVICERs hanging around, and when the time came to start a new one, there would be no slots for new jobs available. When this happened, the EXECUTIVE (job scheduler) would issue a 1201 or 1202 alarm and cause a soft restart of the computer. Every job and task was flushed, and the computer started up fresh, resuming from its last checkpoint. It was essentially a full-on crash and restart, rather than a graceful cancellation of a few jobs. And unlike is often said, the computer wasn't dropping low-priority things; it was failing to complete the most critical job of the landing, the SERVICER. Luckily, the load was light enough that of the SERVICER's duties, the old SERVICER was usually in the final display updating code when it got preempted by a new SERVICER. This caused times in the descent when the display stopped updating entirely, but the flight proceeded mostly as usual. However, with slightly more load, it was fully possible that the SERVICER could have been preempted in the attitude control portion of the code, or worse yet, the throttle control portion. Since each SERVICER shared the same memory location as the last one (since there was only ever supposed to be one running at a time), this could lead to violent attitude or throttle excursions, which would have certainly called for an abort. Luckily, this didn't happen -- and the flight controllers didn't abort the mission not because 1202s were always safe, but because they didn't understand just how bad it could be, were the load just a tiny bit higher." [1] https://news.ycombinator.com/item?id=20791307 https://news.ycombinator.com/item?id=20791307
- ericbarrett 6y ago
  A consumer chip is a lot more likely to be permanently damaged by a "silver bullet" cosmic ray, though. Rad-hard chips don't just have shielding, they can also have redundant circuits and modifications to the foundry process. That said, I'm sure Ingenuity's processor is fit to purpose.
  - alexvoda 6y ago
    It's a bit of a shame that radiation hardened chips are stuck so far in the past. I believe the volume is so low it does not warrant the investment to make radiation hardened versions more often.
    - simonh 6y ago
      Actually being stuck in the past might be a feature. Denser circuitry is likely to be more vulnerable to interference by ionising radiation and more vulnerable to physical damage from high energy particles.
    - numpad0 6y ago
      Parts of hardening include enlarging physical features and increasing operating voltage
  - Dylan16807 6y ago
    > A consumer chip is a lot more likely to be permanently damaged by a "silver bullet" cosmic ray, though. How much more likely? Are the odds all that high over the mission duration?
    - ericbarrett 6y ago
      I don't have any numbers for you, but here's a starting source: https://hps.org/publicinformation/ate/q11162.html https://hps.org/publicinformation/ate/q11162.html Most of my knowledge of the subject is from a friend who did his PhD dissertation on it; specifically, triply-redundant adder circuits for single-bit operations, allowing for some rad-hard chip designs using regular foundry processes.
- inamberclad 6y ago
  Yes, there's a flight qualified microsemi FPGA (a ProASIC, I think) that acts as a supervisor for the vehicle.
m463 6y ago
One thing that's not clear to me is what the radiation is like on mars. Mars has an atmosphere, but no magnetic field. Is it like the situation high in earth's atmosphere (like using your ipad on a commercial flight), or would it be more like on the moon with no protection?
- madaxe_again 6y ago
  Between the two. 30 µSv per hour on the surface of Mars on average compared to 60 µSv on the Moon - compared to 5 µSv on a jet. Not friendly, but not too awful.
  - dmurray 6y ago
    Or 0.5 per hour on Earth at sea level. That's only 2 orders of magnitude difference. Given that we don't expect radiation damage to be a major source of failures in Earthbound consumer or commercial electronics, it's a bit surprising it's such a big deal in space.
    - emkoemko 6y ago
      yes but these chips having to function and survive on the way to the planet right? and they do all sorts of software updates etc during this time. Maybe we could shield them and that would be fine? or would that add to much extra weight
    - nradov 6y ago
      There are actually a lot of bit flips in Earthbound electronics due to lack of parity error checking. But those bit flips don't necessarily cause failures, or when they do it's impossible to determine the root cause.
    - contravariant 6y ago
      Well 2 orders of magnitude can turn a problem that occurs once a decade into a problem that occurs once a month so it's not too surprising that the problem is a bigger deal in space.
    - 6y ago
birktj 6y ago
What I imagine for long lived and high-cost missions is using some sort of co-processor setup with a radiation hardened processor and a faster and more modern processor. These rovers run a lot of computer vision algorithms and I believe more powerful hardware would be quite useful. They may already do something like this, however my understanding is that there is a lot of skepticism in integrating these less fault-tolerant processors. Ingenuity could help remove some of the skepticism and lead to more systems like this in the future.
- alexvoda 6y ago
  I wonder if it wouldn't have been possible to do this on Perseverance too. After Ingenuitys mission is done they could have lugged it arround for extra processing power for as long as it survives. I doubt the extra weight causes that much extra energy consumption. The software stack is probably not ready for that since the plan is to abandon it after its mission. And the Zigbee protocol is rather limmited.
- numpad0 6y ago
  Do keep in mind that radiation cause permanent damages at predictable paces... I don’t understand why online discussions like this fails to address permanent damages. You can’t reset burnt chips.
- GimbalLock 6y ago
  You’re not far off. The Snapdragon is monitored by some more radiation tolerant hardware (FPGA if I recall correctly). Great talk about it from the FSW Workshop in 2019. https://m.youtube.com/watch?v=mQu9m4MG5Gc https://m.youtube.com/watch?v=mQu9m4MG5Gc

Robotbeat 6y ago

It’s less about longevity and more about reducing errors. Interestingly, Ingenuity has some sort of watchdog that can reset the Snapdragon in-flight fast enough to recover if there is an error.

EarthLaunch 6y ago
Reminds me of SpaceX or Erlang; build a process that handles errors rather than an error free process.
- NikolaeVarius 6y ago
  I mean its space. Any computer expected to be in space and do non completely trivial things have tons of mechanisms in place to survive computer issues.
- deleted 6y ago
  [deleted]
ddingus 6y ago
Presumably. That is all part of this test.
jjoonathan 6y ago
Reminds me of Apollo 11, where the computer kept resetting on the way down to the lunar surface but still got Neil Armstrong and Buzz Aldrin to the moon with the whole world watching.
- kabdib 6y ago
  It wasn't resetting, it was ditching lower priority tasks that it didn't have (real) time to accomplish. Notably, the computer was still doing important work (ahem, flying the LM :-) ), and the tasks it didn't have time to do were not critical. The team had recently seen similar failures in simulation, and was able to quickly decide it was okay to proceed.
  - jhayward 6y ago
    > Notably, the computer was still doing important work (ahem, flying the LM :-) ), and the tasks it didn't have time to do were not critical. I believe this is incorrect. From an earlier HN discussion: "The 1202s were also a lot less benign than is often reported. They occurred because of the fixed two-second guidance cycle in the landing software. That is, once every two seconds, a job called the SERVICER would start. SERVICER had many tasks during the landing. In order: navigation, guidance, commanding throttle, commanding attitude, and updating displays. With an excessive load as caused by the CDU, new SERVICERs were starting before old ones could finish. Eventually there would be two many old SERVICERs hanging around, and when the time came to start a new one, there would be no slots for new jobs available. When this happened, the EXECUTIVE (job scheduler) would issue a 1201 or 1202 alarm and cause a soft restart of the computer. Every job and task was flushed, and the computer started up fresh, resuming from its last checkpoint. It was essentially a full-on crash and restart, rather than a graceful cancellation of a few jobs. And unlike is often said, the computer wasn't dropping low-priority things; it was failing to complete the most critical job of the landing, the SERVICER. Luckily, the load was light enough that of the SERVICER's duties, the old SERVICER was usually in the final display updating code when it got preempted by a new SERVICER. This caused times in the descent when the display stopped updating entirely, but the flight proceeded mostly as usual. However, with slightly more load, it was fully possible that the SERVICER could have been preempted in the attitude control portion of the code, or worse yet, the throttle control portion. Since each SERVICER shared the same memory location as the last one (since there was only ever supposed to be one running at a time), this could lead to violent attitude or throttle excursions, which would have certainly called for an abort. Luckily, this didn't happen -- and the flight controllers didn't abort the mission not because 1202s were always safe, but because they didn't understand just how bad it could be, were the load just a tiny bit higher." [1] https://news.ycombinator.com/item?id=20791307 https://news.ycombinator.com/item?id=20791307
ericbarrett 6y ago
A consumer chip is a lot more likely to be permanently damaged by a "silver bullet" cosmic ray, though. Rad-hard chips don't just have shielding, they can also have redundant circuits and modifications to the foundry process. That said, I'm sure Ingenuity's processor is fit to purpose.
- alexvoda 6y ago
  It's a bit of a shame that radiation hardened chips are stuck so far in the past. I believe the volume is so low it does not warrant the investment to make radiation hardened versions more often.
  - simonh 6y ago
    Actually being stuck in the past might be a feature. Denser circuitry is likely to be more vulnerable to interference by ionising radiation and more vulnerable to physical damage from high energy particles.
  - numpad0 6y ago
    Parts of hardening include enlarging physical features and increasing operating voltage
- Dylan16807 6y ago
  > A consumer chip is a lot more likely to be permanently damaged by a "silver bullet" cosmic ray, though. How much more likely? Are the odds all that high over the mission duration?
  - ericbarrett 6y ago
    I don't have any numbers for you, but here's a starting source: https://hps.org/publicinformation/ate/q11162.html https://hps.org/publicinformation/ate/q11162.html Most of my knowledge of the subject is from a friend who did his PhD dissertation on it; specifically, triply-redundant adder circuits for single-bit operations, allowing for some rad-hard chip designs using regular foundry processes.
inamberclad 6y ago
Yes, there's a flight qualified microsemi FPGA (a ProASIC, I think) that acts as a supervisor for the vehicle.

m463 6y ago

One thing that's not clear to me is what the radiation is like on mars. Mars has an atmosphere, but no magnetic field. Is it like the situation high in earth's atmosphere (like using your ipad on a commercial flight), or would it be more like on the moon with no protection?

madaxe_again 6y ago
Between the two. 30 µSv per hour on the surface of Mars on average compared to 60 µSv on the Moon - compared to 5 µSv on a jet. Not friendly, but not too awful.
- dmurray 6y ago
  Or 0.5 per hour on Earth at sea level. That's only 2 orders of magnitude difference. Given that we don't expect radiation damage to be a major source of failures in Earthbound consumer or commercial electronics, it's a bit surprising it's such a big deal in space.
  - emkoemko 6y ago
    yes but these chips having to function and survive on the way to the planet right? and they do all sorts of software updates etc during this time. Maybe we could shield them and that would be fine? or would that add to much extra weight
  - nradov 6y ago
    There are actually a lot of bit flips in Earthbound electronics due to lack of parity error checking. But those bit flips don't necessarily cause failures, or when they do it's impossible to determine the root cause.
  - contravariant 6y ago
    Well 2 orders of magnitude can turn a problem that occurs once a decade into a problem that occurs once a month so it's not too surprising that the problem is a bigger deal in space.
  - 6y ago

birktj 6y ago

What I imagine for long lived and high-cost missions is using some sort of co-processor setup with a radiation hardened processor and a faster and more modern processor. These rovers run a lot of computer vision algorithms and I believe more powerful hardware would be quite useful. They may already do something like this, however my understanding is that there is a lot of skepticism in integrating these less fault-tolerant processors. Ingenuity could help remove some of the skepticism and lead to more systems like this in the future.

alexvoda 6y ago
I wonder if it wouldn't have been possible to do this on Perseverance too. After Ingenuitys mission is done they could have lugged it arround for extra processing power for as long as it survives. I doubt the extra weight causes that much extra energy consumption. The software stack is probably not ready for that since the plan is to abandon it after its mission. And the Zigbee protocol is rather limmited.
numpad0 6y ago
Do keep in mind that radiation cause permanent damages at predictable paces... I don’t understand why online discussions like this fails to address permanent damages. You can’t reset burnt chips.
GimbalLock 6y ago
You’re not far off. The Snapdragon is monitored by some more radiation tolerant hardware (FPGA if I recall correctly). Great talk about it from the FSW Workshop in 2019. https://m.youtube.com/watch?v=mQu9m4MG5Gc https://m.youtube.com/watch?v=mQu9m4MG5Gc