4 ms·

Note that the space shuttle didn't use processors running in lockstep, like the Tandem machines do (from what I understand). It used 5 single-core computers. Fo

Note that the space shuttle didn't use processors running in lockstep, like the Tandem machines do (from what I understand). It used 5 single-core computers. Four of them would run the primary software, with a single control channel controlled by each of the four. For something like the elevons, control was accomplished by connecting 3 actuators to 3 of the channels, with voting being accomplished by physical force - if the 3 computers disagreed, the 2 would overpower the one. (Things like thrusters used electronic voting, close to the thruster itself.) This seems closer to modern architectures with multiple computers, than the mainframe idea of redundant hardware.

Trombone12 10y ago
Note that all those computers shared the same bus, and so that the complete system wasn't as redundant as hoped as this story[1] retells: "At 12:12 GMT 13 May 2008, a space shuttle was loading its hypergolic fuel for mission STS-124 when a 3-1 disagreement occurred among its General Purpose Computers (GPC 4 disagreed with the other GPCs). Three seconds later, the split became 2-1-1 (GPC 2 now disagreed with GPC 4 and the other two GPCs). This required that the launch countdown be stopped. During the subsequent troubleshooting, the remaining two GPCs disagreed (1-1-1-1 split). This was a complete system disagreement. However, none of the GPCs were faulty. The fault was in the FA 2 Multiplexer Demultiplexer. This fault was a crack in a diode. This crack was perpendicular to the normal current flow and completely through the current path. As a crack opened up, it changed the diode into another type of component ... a capacitor. Because some of the bits in the signal are smaller than they should have been, some of the GPC receivers could not see these bits. The ability to see these bits depends on the sensitivity of the receiver, which is a function of manufacturing variances, temperature, and its power supply voltage. From the symptoms, it is apparent that the receiver in GPC 4 was the least sensitive and saw the errors before the other three GPC. This caused GPC 4 to disagree with the other three. Then, as the crack in the diode widened, the bits became shorter to the point where GPC 2 could no longer see these bits; which caused it to disagree with the other GPC. At this point, the set of messages that was received correctly by GPC 4 was different from the set of messages that was correctly received by GPC 2 which was different again from the set of messages that was correctly received by GPC 1 and GPC 3. This process continued until GPC 1 and GPC 3 also disagreed with all the other GPC." [1] Adapted from https://c3.nasa.gov/dashlink/projects/79/wiki/test_stories_split/ https://c3.nasa.gov/dashlink/projects/79/wiki/test_stories_s...
- hashhar 10y ago
  Is there a website which collects/links to such anecdotes and surfaces interesting ones? I would to read more such stories.
ams6110 10y ago
Also I think the processors ran different, independently developed software, the idea being that it would be unlikely for independent codebases to have the same bugs.
- sitharus 10y ago
  The four main CPUs ran the same Primary Flight Software. There was a separate backup computer running the Backup Flight Software, which was developed by an independant team, to take over if the PFS failed. There's a bit of debate as to what would have happened if it ever took over.
  - jackvalentine 10y ago
    I expect they did know, but it'd be an interesting exercise to not tell either development team if they were writing the "real" or the "backup" software, and just swap them around to keep people on their toes.
cjp222 10y ago
Tandem computers did not run software in lockstep to achieve fault tolerance. They were shared-nothing parallel clusters before shared-nothing parallel clusters were cool, with message-based communication between processes independent of the node each was running on. This gave near-linear scalability as the number of nodes grew. I worked on parallel sorting and parallel data loading back in the 1980s, scaling up to 256 nodes, and Tandem's NonStop SQL developed and commercialized similar parallel database query evaluation in the late 1980s, ahead of its time.

Trombone12 10y ago

Note that all those computers shared the same bus, and so that the complete system wasn't as redundant as hoped as this story[1] retells: "At 12:12 GMT 13 May 2008, a space shuttle was loading its hypergolic fuel for mission STS-124 when a 3-1 disagreement occurred among its General Purpose Computers (GPC 4 disagreed with the other GPCs). Three seconds later, the split became 2-1-1 (GPC 2 now disagreed with GPC 4 and the other two GPCs). This required that the launch countdown be stopped. During the subsequent troubleshooting, the remaining two GPCs disagreed (1-1-1-1 split). This was a complete system disagreement. However, none of the GPCs were faulty. The fault was in the FA 2 Multiplexer Demultiplexer. This fault was a crack in a diode. This crack was perpendicular to the normal current flow and completely through the current path. As a crack opened up, it changed the diode into another type of component ... a capacitor. Because some of the bits in the signal are smaller than they should have been, some of the GPC receivers could not see these bits. The ability to see these bits depends on the sensitivity of the receiver, which is a function of manufacturing variances, temperature, and its power supply voltage. From the symptoms, it is apparent that the receiver in GPC 4 was the least sensitive and saw the errors before the other three GPC. This caused GPC 4 to disagree with the other three. Then, as the crack in the diode widened, the bits became shorter to the point where GPC 2 could no longer see these bits; which caused it to disagree with the other GPC. At this point, the set of messages that was received correctly by GPC 4 was different from the set of messages that was correctly received by GPC 2 which was different again from the set of messages that was correctly received by GPC 1 and GPC 3. This process continued until GPC 1 and GPC 3 also disagreed with all the other GPC." [1] Adapted from https://c3.nasa.gov/dashlink/projects/79/wiki/test_stories_split/ https://c3.nasa.gov/dashlink/projects/79/wiki/test_stories_s...

hashhar 10y ago
Is there a website which collects/links to such anecdotes and surfaces interesting ones? I would to read more such stories.

ams6110 10y ago

Also I think the processors ran different, independently developed software, the idea being that it would be unlikely for independent codebases to have the same bugs.

sitharus 10y ago
The four main CPUs ran the same Primary Flight Software. There was a separate backup computer running the Backup Flight Software, which was developed by an independant team, to take over if the PFS failed. There's a bit of debate as to what would have happened if it ever took over.
- jackvalentine 10y ago
  I expect they did know, but it'd be an interesting exercise to not tell either development team if they were writing the "real" or the "backup" software, and just swap them around to keep people on their toes.

cjp222 10y ago

Tandem computers did not run software in lockstep to achieve fault tolerance. They were shared-nothing parallel clusters before shared-nothing parallel clusters were cool, with message-based communication between processes independent of the node each was running on. This gave near-linear scalability as the number of nodes grew. I worked on parallel sorting and parallel data loading back in the 1980s, scaling up to 256 nodes, and Tandem's NonStop SQL developed and commercialized similar parallel database query evaluation in the late 1980s, ahead of its time.