6 ms·

This is the kind of failure mode that I imagine (now with hindsight) should be mitigated by (re)designing the system in such a way that the timestamp desync wou

by tgbugs 5y ago

This is the kind of failure mode that I imagine (now with hindsight) should be mitigated by (re)designing the system in such a way that the timestamp desync would always be detected because each part of the system should have its own internal model of the approximate times that it should be receiving from other systems and never blindly trust them. I imagine it is sort of like the organisms early in evolution that simply believed their sensory system when it had in fact been hijacked by something that wanted to eat them. There aren't many sensory systems with those issues these days because anything that doesn't have an internal model that it can use for comparison winds up being a snack or crashing on an alien planet.

SV_BubbleTime 5y ago
I struggle to think that JPL didn’t think of this already. That in the distance/time code they didn’t take timestamps of each successful frame vs just assuming it’s a 30hz camera so surely each frame will be 30hz! It seems to me, as someone who does similar embedded work, that if I would account for a missing frame failure, they surely would. I am a professional in embedded timing systems - I am guessing this isn’t the full story or it’s been paraphrased for normal people and something important was lost. EDIT: Better info below. The photo was lost but it’s timestamp was not. They apparently didn’t tightly couple the image and it’s metadata together (hashes... what are they for!?) so there was some sort of mess up there. Knew there was more to the story.
- dmurray 5y ago
  I wonder if hashes are discouraged at JPL due to a historical focus on "correct, and provably correct" code. Hash collisions happen. Yes, you can make collisions arbitrarily unlikely, and there are so many situations where they come in useful, but I can imagine an engineering culture where they are just never an easy tool to reach for.
  - dnautics 5y ago
    You can axiomatize uniqueness of hashes in proof systems.
    - ethbr0 5y ago
      This would require a hash input (or combination of inputs) to be unique over a limited / expected domain, right?
      - ncallaway 5y ago
        I mean, you could take anything as an axiom so there are no real requirements for it. I would imagine you would want to demonstrate the P(collision|domain+operating lifetime) << acceptable level or risk, and after you’d done that, take the uniqueness of hashes as a axiom for the proof system? That’s just speculation on my part, since I haven’t done much with formal proofs of working code
      - dnautics 5y ago
        No you just special case the function call and tell the proof analyzer "this never happens". Strictly speaking, a lie, but probably a sophisticated prover could even be rigged to keep track of those conditions (not in x years, e.g.) if you so chose
  - SV_BubbleTime 5y ago
    > arbitrarily unlikely IDK, I would call more possibilities than grains of sand in the universe a little more than “arbitrarily unlikely”.
- twhitmore 5y ago
  It's not actually a time "stamp" if you don't keep it on or tightly associated.. A struct or pair of {Image, Timestamp} would be the simplest & most robust solution.
- richardfey 5y ago
  Exactly my thoughts. This is something I would expect covered by the testing process. I wonder what the Apollo engineers would say about this.
tremon 5y ago
We'll have to wait and see if they publish a public post-mortem, but this is a textbook real-time scheduling problem and it surprises me. Scheduling theory teaches to abandon processing (a frame/image in this case) if the result is late, both to avoid doing unnecessary work and to recover from sync/latency issues. From the minimal descriptions I've seen so far, it seems like they were using producer/consumer work queues for the image processing without further synchronization.
- widforss 5y ago
  Say we would assume this is a complete real-time system. How would a result be late? It seems like a failure mode that doesn't exist in that domain (given that all guarantees are verified). Now, I get that this is probably not a real-time system.
  - jannes 5y ago
    It’s the first time they are using commodity hardware on mars. I believe the CPU is the same that has been used in a few smartphones. So it’s not a real-time system. https://9to5google.com/2021/04/19/nasa-ingenuity-qualcomm-snapdragon/ https://9to5google.com/2021/04/19/nasa-ingenuity-qualcomm-sn...
    - siftedpixel 5y ago
      Can’t you have a real-time kernel?
    - NovemberWhiskey 5y ago
      There is no single CPU; it's a multitiered control system with at least two of the tiers automotive/military grade components for realtime systems. ref. https://news.ycombinator.com/item?id=26907669 https://news.ycombinator.com/item?id=26907669 for more details for example.
    - _joel 5y ago
      There's 2 copies of memory and a RAD hardened FPGA that compares them and looks for bitflips. If it detects one then it restarts, so quickly it can do this midflight with no issues.
loa_in_ 5y ago
That sounds nearly impossible a feature to implement. Simulating a virtual copy of a component that's purpose is to deliver novel data, autonomously?
- timmattison 5y ago
  I think it can be simpler than that. If a piece of code is going to do a velocity calculation from motion estimation between two frames it should expect to get new data every 1 / FPS seconds. A local clock could help validate that. If data comes too fast or too slow within some tolerance then the system could signal there is an issue. If the images have time stamps it could check those too. Now you’d need to hope the inertial guidance is good enough at those altitudes. The article says it doesn’t use image based calculations for landing due to the possibility of kicking up dust.
  - zoomablemind 5y ago
    > "...This glitch caused a single image to be lost, but more importantly, it resulted in all later navigation images being delivered with inaccurate timestamps..." It seems that the timestamping is detached from frameshooting. That is, as if there were a burst happening, then the burst's frames get assigned the sequential timestamps. Thus if a burst had a glitched/skipped frame, a part of the series and the subsequent bursts become mis-attributed. I understand that integrating the timestamping to be atomic to frameshooting is likely to slowdown the burst. So there has to be some expected internal timeline to validate the timeline that results from bursting.
crocal 5y ago
This is actually compulsory requirement for safe communication protocols in railway according to EN 50159. The correct way to manage this is to reject the outdated samples. Ultimately, this should result in a transition to fallback mode (in this case, emergency landing mode).
- GeorgeTirebiter 5y ago
  This is actually all very complicated, because if you're doing e.g. velocity estimation via smoothed differences, missing a point means you've (effectively) inserted a spike-like pseudo-datapoint into your Kalman filter (or whatever) --- it will not totally crap, but will be incorrect until the 'bad' datapoint flows completely through the filter.
  - crocal 5y ago
    ... Indeed. Therefore we add sensor diversity to balance the failure modes and increase availability. Having no sample means that you increase your confidence interval at warp speed to account for all possible scenarios. Very quickly your estimate becomes : « great, I am traveling at 35kph +/- 100kph. Wait. What? »
SamBam 5y ago
> I imagine it is sort of like the organisms ... Can you describe this further, or link to further reading?
- tgbugs 5y ago
  I don't have anything specific in mind that covers this (I'll look around), but maybe take a look at the work by Nicholas Strausfeld for some actual data on early nervous systems [0], and Leigh van Valen for the general theoretical framework [1]. 0. https://scholar.google.com/scholar?q=Nicholas+Strausfeld https://scholar.google.com/scholar?q=Nicholas+Strausfeld 1. https://www.mn.uio.no/cees/english/services/van-valen/evolutionary-theory/ https://www.mn.uio.no/cees/english/services/van-valen/evolut...

by tgbugs 5y ago

SV_BubbleTime 5y ago

I struggle to think that JPL didn’t think of this already. That in the distance/time code they didn’t take timestamps of each successful frame vs just assuming it’s a 30hz camera so surely each frame will be 30hz! It seems to me, as someone who does similar embedded work, that if I would account for a missing frame failure, they surely would. I am a professional in embedded timing systems - I am guessing this isn’t the full story or it’s been paraphrased for normal people and something important was lost. EDIT: Better info below. The photo was lost but it’s timestamp was not. They apparently didn’t tightly couple the image and it’s metadata together (hashes... what are they for!?) so there was some sort of mess up there. Knew there was more to the story.

dmurray 5y ago
I wonder if hashes are discouraged at JPL due to a historical focus on "correct, and provably correct" code. Hash collisions happen. Yes, you can make collisions arbitrarily unlikely, and there are so many situations where they come in useful, but I can imagine an engineering culture where they are just never an easy tool to reach for.
- dnautics 5y ago
  You can axiomatize uniqueness of hashes in proof systems.
  - ethbr0 5y ago
    This would require a hash input (or combination of inputs) to be unique over a limited / expected domain, right?
    - ncallaway 5y ago
      I mean, you could take anything as an axiom so there are no real requirements for it. I would imagine you would want to demonstrate the P(collision|domain+operating lifetime) << acceptable level or risk, and after you’d done that, take the uniqueness of hashes as a axiom for the proof system? That’s just speculation on my part, since I haven’t done much with formal proofs of working code
    - dnautics 5y ago
      No you just special case the function call and tell the proof analyzer "this never happens". Strictly speaking, a lie, but probably a sophisticated prover could even be rigged to keep track of those conditions (not in x years, e.g.) if you so chose
- SV_BubbleTime 5y ago
  > arbitrarily unlikely IDK, I would call more possibilities than grains of sand in the universe a little more than “arbitrarily unlikely”.
twhitmore 5y ago
It's not actually a time "stamp" if you don't keep it on or tightly associated.. A struct or pair of {Image, Timestamp} would be the simplest & most robust solution.
richardfey 5y ago
Exactly my thoughts. This is something I would expect covered by the testing process. I wonder what the Apollo engineers would say about this.

tremon 5y ago

We'll have to wait and see if they publish a public post-mortem, but this is a textbook real-time scheduling problem and it surprises me. Scheduling theory teaches to abandon processing (a frame/image in this case) if the result is late, both to avoid doing unnecessary work and to recover from sync/latency issues. From the minimal descriptions I've seen so far, it seems like they were using producer/consumer work queues for the image processing without further synchronization.

widforss 5y ago
Say we would assume this is a complete real-time system. How would a result be late? It seems like a failure mode that doesn't exist in that domain (given that all guarantees are verified). Now, I get that this is probably not a real-time system.
- jannes 5y ago
  It’s the first time they are using commodity hardware on mars. I believe the CPU is the same that has been used in a few smartphones. So it’s not a real-time system. https://9to5google.com/2021/04/19/nasa-ingenuity-qualcomm-snapdragon/ https://9to5google.com/2021/04/19/nasa-ingenuity-qualcomm-sn...
  - siftedpixel 5y ago
    Can’t you have a real-time kernel?
  - NovemberWhiskey 5y ago
    There is no single CPU; it's a multitiered control system with at least two of the tiers automotive/military grade components for realtime systems. ref. https://news.ycombinator.com/item?id=26907669 https://news.ycombinator.com/item?id=26907669 for more details for example.
  - _joel 5y ago
    There's 2 copies of memory and a RAD hardened FPGA that compares them and looks for bitflips. If it detects one then it restarts, so quickly it can do this midflight with no issues.

loa_in_ 5y ago

That sounds nearly impossible a feature to implement. Simulating a virtual copy of a component that's purpose is to deliver novel data, autonomously?

timmattison 5y ago
I think it can be simpler than that. If a piece of code is going to do a velocity calculation from motion estimation between two frames it should expect to get new data every 1 / FPS seconds. A local clock could help validate that. If data comes too fast or too slow within some tolerance then the system could signal there is an issue. If the images have time stamps it could check those too. Now you’d need to hope the inertial guidance is good enough at those altitudes. The article says it doesn’t use image based calculations for landing due to the possibility of kicking up dust.
- zoomablemind 5y ago
  > "...This glitch caused a single image to be lost, but more importantly, it resulted in all later navigation images being delivered with inaccurate timestamps..." It seems that the timestamping is detached from frameshooting. That is, as if there were a burst happening, then the burst's frames get assigned the sequential timestamps. Thus if a burst had a glitched/skipped frame, a part of the series and the subsequent bursts become mis-attributed. I understand that integrating the timestamping to be atomic to frameshooting is likely to slowdown the burst. So there has to be some expected internal timeline to validate the timeline that results from bursting.

crocal 5y ago

This is actually compulsory requirement for safe communication protocols in railway according to EN 50159. The correct way to manage this is to reject the outdated samples. Ultimately, this should result in a transition to fallback mode (in this case, emergency landing mode).

GeorgeTirebiter 5y ago
This is actually all very complicated, because if you're doing e.g. velocity estimation via smoothed differences, missing a point means you've (effectively) inserted a spike-like pseudo-datapoint into your Kalman filter (or whatever) --- it will not totally crap, but will be incorrect until the 'bad' datapoint flows completely through the filter.
- crocal 5y ago
  ... Indeed. Therefore we add sensor diversity to balance the failure modes and increase availability. Having no sample means that you increase your confidence interval at warp speed to account for all possible scenarios. Very quickly your estimate becomes : « great, I am traveling at 35kph +/- 100kph. Wait. What? »

SamBam 5y ago

> I imagine it is sort of like the organisms ... Can you describe this further, or link to further reading?

tgbugs 5y ago
I don't have anything specific in mind that covers this (I'll look around), but maybe take a look at the work by Nicholas Strausfeld for some actual data on early nervous systems [0], and Leigh van Valen for the general theoretical framework [1]. 0. https://scholar.google.com/scholar?q=Nicholas+Strausfeld https://scholar.google.com/scholar?q=Nicholas+Strausfeld 1. https://www.mn.uio.no/cees/english/services/van-valen/evolutionary-theory/ https://www.mn.uio.no/cees/english/services/van-valen/evolut...