6 ms·

This makes no sense in the way it is stated - the calculations involved are independent of the up-time. You do not aim at a target differently depending on how

by danbruc 13y ago

This makes no sense in the way it is stated - the calculations involved are independent of the up-time. You do not aim at a target differently depending on how long the system is up. The correct way to think about this is probably the following - they noticed that the up-time drifted and tried to improve on that, but they failed to do so in all places. In consequence different parts of the system used different times that also advanced with differing speeds and this inconsistencies caused the calculations to go wrong. As mentioned in the article, would they not have made the "improvement" all parts would have used the same time and the errors would have canceled because the calculations are time-independent and the drift is probably to small to be important during the relatively short time a target approaches. This seems one of the rare cases where blaming math with limited precision is wrong.

fennecfoxen 13y ago
> This makes no sense in the way it is stated - the calculations involved are independent of the up-time. You do not aim at a target differently depending on how long the system is up. At some point or another your anti-SCUD computer is going to have to say "Send the electrical signal to launch this missile NOW!". I assume this is where they make use of the system clock.
- danbruc 13y ago
  Definitely, but it does not matter what the value of NOW is as long as all parts use the same time. Only their attempt to fix the drift caused different parts to have a different understanding what NOW = 5484515 means and that caused the problem.
  - deleted 13y ago
    [deleted]
jessaustin 13y ago
Do you have additional information about this incident? Although TFA alludes to something like what you're saying with its "inaccuracies did not cancel" comment, your point seems to require an assumption that is not stated there. Aren't you assuming a non-distributed system, when it seems very likely that radar sensors and missile launchers would not be co-located? In a distributed system, we certainly cannot assume identical boot dates, and so an accumulating error would accumulate differently at different components of the system. Systems like this must have time agreement among their different components. It seems that a system developed later would have just used GPS to prevent this problem, but I wonder why they didn't use NTP?
- danbruc 13y ago
  The obvious evidence are the laws of physics which are invariant under time translation. What you are saying is exactly what I said - it does not matter if the drifting clocks are in different parts of a distribute system or if a single system uses different clocks. In the end the system fails because different clocks disagree on the current time, not because they drift away from the actual time. And the article states exactly this - they tried to fix the clock drift relative to the actual time but failed to do so in all components and thereby inadvertently introduced two clocks drifting relative to each other.
  - Varcht 13y ago
    Yep, that's how I read the article, the target path prediction system was 0.34 seconds out of sync with the targeting system due to differing implementations.
  - jessaustin 13y ago
    In the end the system fails because different clocks disagree on the current time, not because they drift away from the actual time. The possibility I mentioned was that the clocks disagreed because they had drifted away from actual time at similar rates, for different periods. [r ≠ 1 ∧ t₀ ≠ t₁] → [t₀ + (t-t₀)r ≠ t₁ + (t-t₁)r]. Unless specifically addressed, this phenomenon will occur in distributed systems. That said, the link 'gibrown provided seems to establish firmly that this particular error resulted from times being up-converted (24bit fixed to 48bit floating) via different methods in different parts of the system.
    - danbruc 13y ago
      Now I see what you mean, equal drift rates but different up-time. Assuming two clocks coming up at different points in time this will indeed cause time differences if they are initially synchronized with a third clock. If the clock coming up later synchronizes with the other clock already running there should be no issues though. I read the article, too, and this two different ways of converting are more or less equivalent of two clocks starting in sync but drifting relative to each other. Therefore my initial comment was quite on the spot.
msmillie 13y ago
The following scenario helps explain: you have two radars. One is a wide-angle, general radar, which sees a missle at some coordinates, travelling at some speed. The other radar is for target acquisition, has a very narrow spread ("range gate"), and must determine the precise location of the missle given an initial set of coordinates, a velocity, and a timestamp. In other words, the system is actually aiming at targets differently depending on how long it has been up, tracking so far in front (or behind, I don't recall the details) that it can't acquire the target.
- danbruc 13y ago
  You are describing a problem where two clocks drift relative to each other, not a problem where one clock drifts away from the actual time. On the other hand the article gives the impression the failure occurred because the system failed to exactly measure the up-time, not because clocks in different systems or system components drifted relative to each other.
  - vonmoltke 13y ago
    That is not the impression I got from the article. No logic in the system cares how long it has been up, not directly at least. What matters is drift from its time reference, which is a function of uptime. Various modules in a complex system like this each have their own clock, which I will refer to generically as a real-time binary counter (RTBC), which the module uses as its event time reference. The RTBC starts at 0 when the module comes up. At some point shortly after coming up the module will check in with its controller, which will send a time-of-day (TOD) message. The module links the TOD message to a particular RTBC tick to create its time reference. At this point the time is free to start drifting relative to the actual wall clock time, until the system is power cycled again.
    - danbruc 13y ago
      That is exactly what I said - different clocks drifting relative to each other. It is completely irrelevant that their one tenth of a second was not exactly one tenth of a second, what matters is that different clocks in the system had different ideas of one tenth of a second.
salem 13y ago
My understanding is that they used a floating point value for time measurements, and they used time since boot as a reference. After a long enough time since boot, there were not enough significant digits available to represent both time since boot, and accurately capture the different in time between two events. The first Ariane 5 launch also had a failure relating to floating point math, in that case an overflow when converting to an int16: http://www.di.unito.it/~damiani/ariane5rep.html http://www.di.unito.it/~damiani/ariane5rep.html
- danbruc 13y ago
  The article states they use a 24 bit fixed point format.
  - salem 13y ago
    Sure, but the point is that it did not have enough precision for both a large absolute value, and small time measurements, when based on time since boot.
    - danbruc 13y ago
      No, this is not what happened. They converted the 24 bit fixed point value into a different format using different algorithms in different places of the system resulting in slightly different values. The failure was then caused by the difference between these values, i.e. different parts of the system did not agree on what the current time is.
gibrown 13y ago
A better description of the bug: http://www.ual.es/~plopez/docencia/itis/patriot.htm http://www.ual.es/~plopez/docencia/itis/patriot.htm It was related to limited precision, but was also related to using different calculation methods in different parts of the code.

by danbruc 13y ago

fennecfoxen 13y ago

> This makes no sense in the way it is stated - the calculations involved are independent of the up-time. You do not aim at a target differently depending on how long the system is up. At some point or another your anti-SCUD computer is going to have to say "Send the electrical signal to launch this missile NOW!". I assume this is where they make use of the system clock.

danbruc 13y ago
Definitely, but it does not matter what the value of NOW is as long as all parts use the same time. Only their attempt to fix the drift caused different parts to have a different understanding what NOW = 5484515 means and that caused the problem.
- deleted 13y ago
  [deleted]

jessaustin 13y ago

Do you have additional information about this incident? Although TFA alludes to something like what you're saying with its "inaccuracies did not cancel" comment, your point seems to require an assumption that is not stated there. Aren't you assuming a non-distributed system, when it seems very likely that radar sensors and missile launchers would not be co-located? In a distributed system, we certainly cannot assume identical boot dates, and so an accumulating error would accumulate differently at different components of the system. Systems like this must have time agreement among their different components. It seems that a system developed later would have just used GPS to prevent this problem, but I wonder why they didn't use NTP?

danbruc 13y ago
The obvious evidence are the laws of physics which are invariant under time translation. What you are saying is exactly what I said - it does not matter if the drifting clocks are in different parts of a distribute system or if a single system uses different clocks. In the end the system fails because different clocks disagree on the current time, not because they drift away from the actual time. And the article states exactly this - they tried to fix the clock drift relative to the actual time but failed to do so in all components and thereby inadvertently introduced two clocks drifting relative to each other.
- Varcht 13y ago
  Yep, that's how I read the article, the target path prediction system was 0.34 seconds out of sync with the targeting system due to differing implementations.
- jessaustin 13y ago
  In the end the system fails because different clocks disagree on the current time, not because they drift away from the actual time. The possibility I mentioned was that the clocks disagreed because they had drifted away from actual time at similar rates, for different periods. [r ≠ 1 ∧ t₀ ≠ t₁] → [t₀ + (t-t₀)r ≠ t₁ + (t-t₁)r]. Unless specifically addressed, this phenomenon will occur in distributed systems. That said, the link 'gibrown provided seems to establish firmly that this particular error resulted from times being up-converted (24bit fixed to 48bit floating) via different methods in different parts of the system.
  - danbruc 13y ago
    Now I see what you mean, equal drift rates but different up-time. Assuming two clocks coming up at different points in time this will indeed cause time differences if they are initially synchronized with a third clock. If the clock coming up later synchronizes with the other clock already running there should be no issues though. I read the article, too, and this two different ways of converting are more or less equivalent of two clocks starting in sync but drifting relative to each other. Therefore my initial comment was quite on the spot.

msmillie 13y ago

The following scenario helps explain: you have two radars. One is a wide-angle, general radar, which sees a missle at some coordinates, travelling at some speed. The other radar is for target acquisition, has a very narrow spread ("range gate"), and must determine the precise location of the missle given an initial set of coordinates, a velocity, and a timestamp. In other words, the system is actually aiming at targets differently depending on how long it has been up, tracking so far in front (or behind, I don't recall the details) that it can't acquire the target.

danbruc 13y ago
You are describing a problem where two clocks drift relative to each other, not a problem where one clock drifts away from the actual time. On the other hand the article gives the impression the failure occurred because the system failed to exactly measure the up-time, not because clocks in different systems or system components drifted relative to each other.
- vonmoltke 13y ago
  That is not the impression I got from the article. No logic in the system cares how long it has been up, not directly at least. What matters is drift from its time reference, which is a function of uptime. Various modules in a complex system like this each have their own clock, which I will refer to generically as a real-time binary counter (RTBC), which the module uses as its event time reference. The RTBC starts at 0 when the module comes up. At some point shortly after coming up the module will check in with its controller, which will send a time-of-day (TOD) message. The module links the TOD message to a particular RTBC tick to create its time reference. At this point the time is free to start drifting relative to the actual wall clock time, until the system is power cycled again.
  - danbruc 13y ago
    That is exactly what I said - different clocks drifting relative to each other. It is completely irrelevant that their one tenth of a second was not exactly one tenth of a second, what matters is that different clocks in the system had different ideas of one tenth of a second.

salem 13y ago

My understanding is that they used a floating point value for time measurements, and they used time since boot as a reference. After a long enough time since boot, there were not enough significant digits available to represent both time since boot, and accurately capture the different in time between two events. The first Ariane 5 launch also had a failure relating to floating point math, in that case an overflow when converting to an int16: http://www.di.unito.it/~damiani/ariane5rep.html http://www.di.unito.it/~damiani/ariane5rep.html

danbruc 13y ago
The article states they use a 24 bit fixed point format.
- salem 13y ago
  Sure, but the point is that it did not have enough precision for both a large absolute value, and small time measurements, when based on time since boot.
  - danbruc 13y ago
    No, this is not what happened. They converted the 24 bit fixed point value into a different format using different algorithms in different places of the system resulting in slightly different values. The failure was then caused by the difference between these values, i.e. different parts of the system did not agree on what the current time is.

gibrown 13y ago

A better description of the bug: http://www.ual.es/~plopez/docencia/itis/patriot.htm http://www.ual.es/~plopez/docencia/itis/patriot.htm It was related to limited precision, but was also related to using different calculation methods in different parts of the code.