4 ms·

The devops fix? Reboot every night. Works wonders. :/

by bronson 13y ago

angersock 13y ago
That's an interesting thing about system design philosophy, right? If you just accept that there are unknown unknowns, and make the system easy to fix and bring back up, you can oftentimes do better than a perfectly designed architecture (which matches its model of reality well but not reality itself). Similar engineering feats would be the inclusion of the bolt-assist on the M16 family of rifles and the design of Erlang.
jessaustin 13y ago
That's the first thing I thought, but I wonder if it would violate SLAs to have the system down for a couple of minutes on a daily basis? In this case SLA violations could be penalized severely.
- arbitrage 13y ago
  They probably reboot them in batches, so not all of them were unavailable at the same time.
salem 13y ago
This is exactly what the Israelis figured out and told the Pentagon before this failed interception, but the information was not passed on to units in Saudi Arabia.
- snacktimetoday 13y ago
  The information was not passed on because it was damaging to the military industrial complex interests at the time. Americans at that time in particular really felt infallible after the Cold War and did not accept feedback very well. The skunkworks/American military machine was starting to crack apart with laziness as things descending into contracting hell. As I mentioned in my other comment, my perspective as an Israeli witnessing all this was, "Don't bite the hand that feeds you." It is a shame that Americans and others had to pay a price for arrogance and politics. It may not be the official story, but anyone exposed to that world could tell you otherwise.
  - salem 13y ago
    They did however release a software patch within about 2 weeks of getting the report from Israel, but it took 10 days to make it to Saudi Arabia, one day late.
    - snacktimetoday 13y ago
      In time of war, minutes are an eternity. No reason these soldiers should have had to wait so long other than inefficiency.
chiph 13y ago
That's actually what they did as an interim measure, once the error was identified.
LandoCalrissian 13y ago
I actually think reboots in military hardware like this are pretty common, the way you only have to debug a certain time window.
draker 13y ago
Reboot three times each day; though the Army assumes you know that. "Army officials said that they believed the Israeli experience was atypical--they assumed other Patriot users were not running their systems for 8 or more hours at a time. However, after analyzing the Israeli data and confirming some loss in targeting accuracy, the officials made a software change which compensated for the inaccurate time calculation. This change allowed for extended run times and was included in the modified software version that was released on February 16, 1991. However, Army officials did not use the Israeli data to determine how long the Patriot could operate before the inaccurate time calculation would render the system ineffective. On February 21, 1991, the Patriot Project Office sent a message to Patriot users stating that very long run times could cause a shift in the range gate, resulting in the target being offset. The message also said a software change was being sent that would improve the system's targeting. However, the message did not specify what constitutes very long run times. According to Army officials, they presumed that the users would not continuously run the batteries for such extended periods of time that the Patriot would fail to track targets. Therefore, they did not think that more detailed guidance was required." http://www.fas.org/spp/starwars/gao/im92026.htm http://www.fas.org/spp/starwars/gao/im92026.htm

The devops fix? Reboot every night. Works wonders. :/

by bronson 13y ago

The devops fix? Reboot every night. Works wonders. :/

That's an interesting thing about system design philosophy, right? If you just accept that there are unknown unknowns, and make the system easy to fix and bring back up, you can oftentimes do better than a perfectly designed architecture (which matches its model of reality well but not reality itself). Similar engineering feats would be the inclusion of the bolt-assist on the M16 family of rifles and the design of Erlang.

jessaustin 13y ago

That's the first thing I thought, but I wonder if it would violate SLAs to have the system down for a couple of minutes on a daily basis? In this case SLA violations could be penalized severely.

arbitrage 13y ago
They probably reboot them in batches, so not all of them were unavailable at the same time.

salem 13y ago

This is exactly what the Israelis figured out and told the Pentagon before this failed interception, but the information was not passed on to units in Saudi Arabia.

snacktimetoday 13y ago
The information was not passed on because it was damaging to the military industrial complex interests at the time. Americans at that time in particular really felt infallible after the Cold War and did not accept feedback very well. The skunkworks/American military machine was starting to crack apart with laziness as things descending into contracting hell. As I mentioned in my other comment, my perspective as an Israeli witnessing all this was, "Don't bite the hand that feeds you." It is a shame that Americans and others had to pay a price for arrogance and politics. It may not be the official story, but anyone exposed to that world could tell you otherwise.
- salem 13y ago
  They did however release a software patch within about 2 weeks of getting the report from Israel, but it took 10 days to make it to Saudi Arabia, one day late.
  - snacktimetoday 13y ago
    In time of war, minutes are an eternity. No reason these soldiers should have had to wait so long other than inefficiency.

chiph 13y ago

That's actually what they did as an interim measure, once the error was identified.

LandoCalrissian 13y ago

I actually think reboots in military hardware like this are pretty common, the way you only have to debug a certain time window.

draker 13y ago

Reboot three times each day; though the Army assumes you know that. "Army officials said that they believed the Israeli experience was atypical--they assumed other Patriot users were not running their systems for 8 or more hours at a time. However, after analyzing the Israeli data and confirming some loss in targeting accuracy, the officials made a software change which compensated for the inaccurate time calculation. This change allowed for extended run times and was included in the modified software version that was released on February 16, 1991. However, Army officials did not use the Israeli data to determine how long the Patriot could operate before the inaccurate time calculation would render the system ineffective. On February 21, 1991, the Patriot Project Office sent a message to Patriot users stating that very long run times could cause a shift in the range gate, resulting in the target being offset. The message also said a software change was being sent that would improve the system's targeting. However, the message did not specify what constitutes very long run times. According to Army officials, they presumed that the users would not continuously run the batteries for such extended periods of time that the Patriot would fail to track targets. Therefore, they did not think that more detailed guidance was required." http://www.fas.org/spp/starwars/gao/im92026.htm http://www.fas.org/spp/starwars/gao/im92026.htm