4 ms·

This reminds me of a presentation by Twitter at a conference where they discussed MTBR (Mean Time Before Recovery) instead of MTBF (Mean Time Before Failure).

This reminds me of a presentation by Twitter at a conference where they discussed MTBR (Mean Time Before Recovery) instead of MTBF (Mean Time Before Failure). Instead of trying to prevent all possible failures, they focused on acceptable failure and speedy recovery. If a server takes twice as long to response to a request, mark it offline and let the automated recovery process diagnose the problem and if it's just an anomaly mark it back online. It was pretty interesting and we took a good look at our architecture and found ways to better cater to acceptable failures.

there 14y ago
Audi's strategy in the early 2000s for winning Le Mans 12- and 24-hour endurance races against Bentley was to engineer their cars around failure. The Bentley cars were faster, but Audi engineered theirs to be easier to fix. Over such long races, there were bound to be crashes or failures of various parts and the faster Audi could fix their cars, the quicker they could get back out on the track to outlast the faster cars. A great example of this was that Bentley used the lightest bolt they could everywhere on the car to minimize weight, but Audi used the same size bolt everywhere to minimize the number of tool changes needed to replace the parts.
WalterBright 14y ago
Airliners are designed to withstand failure, not to not have any failures. It's a change in perspective, and is very effective. Both the Fukishima plant and the Deep Water Horizon rig were unable to withstand single failures.
- Mvandenbergh 14y ago
  Nuclear power plants and oil rigs are also designed to withstand failure.
- GoodIntentions 14y ago
  >> Fukishima plant Actually it sounds like the plant failed due to some poor - in retrospect - design assumptions. The generators required to keep cooling systems running safely were buggered by the same disaster. "fail-over" - they got it half right.
  - WalterBright 14y ago
    The problem is when the backup system is not independent of the primary, so one failure takes out the other. When I read about the step-by-step domino sequence of failures in the plant, the design was clearly deficient. The erroneous thinking was "the tsunami wall cannot fail". The correct thinking is "what happens when the tsunami wall is breached? what happens to the plant? how can the plant withstand that?" For example, the most obvious thing is to harden the backup diesel generators against flooding. This would not have been difficult nor expensive. Another solution would have been to have those generators located at some distance from the plant, so they could be repaired without irradiating the workers, and so a disaster at one would be less likely to affect the other. Another obviously poor design was to have the emergency hydrogen venting go into an enclosed space, where it can build up and explode, rather than venting it outside.
    - abduhl 14y ago
      I don't know what your background is but, based on your post, I am assuming it is not in civil or mechanical engineering. Please excuse me if I'm wrong. The design of this power plant probably went like most designs in civil engineering where extreme design loadings were defined (500-year quake, 100-year flood, etc.) and the plant was designed to meet the loads that would be imposed by the most critical event. For a critical piece of infrastructure these loading conditions are quite stringent. While these loadings drive the design at one end, budget drives the design at the other. Most civil projects tend to be just safe enough and not more. This leads to the realm of acceptable risk/failure. The tsunami wall failing under a 1000-year quake is an acceptable failure. The backup systems were most likely designed for progressively lower degrees of acceptable failure. At some point though it becomes too costly to over-design. Was hardening the backup generators against flooding "not difficult nor expensive"? I can't tell you that, I didn't design it. Based on the fact that they weren't hardened, I would say that it probably was difficult or expensive. Would placing the backup generators off site be a better idea than having them on site? Possibly. Possibly not. Every meter you move the generators away from the plant is another meter's worth of risk. How many redundant transmission lines do you need for a generator a mile away? How many redundant generators do you need now that you have the added risk of possibly losing a generator's transmission lines? Contrary to common belief, public works projects do not have an infinite budget. On top of this, working with municipalities generally results in ridiculous amounts of regulations regarding pricing, reviews, and permitting. While this maintains a minimum standard for designs, it also drives all designs down to that minimum standard. It is simply too costly to continually design for more and more unlikely events. Users of this site should be well familiar with this based on the number of Internet Explorer compatability comments I've read lately.
- chriszf 14y ago
  Actually, nuclear power plants are designed to withstand failure. Typically there are multiple backup plans to safeguard against failure of any one system. One comment I sometimes hear is, "They had to pump seawater in to cool things down? That's crazy! They should have had a backup plan." It turns out that nuclear plants are intentionally built near bodies of water because external cooling is a backup, should the primary cooling systems fail. Nothing, however, is designed to withstand systematic catastrophic failure of all of your backup plans.
- lloeki 14y ago
  > the Fukishima plant ... were unable to withstand single failures. I don't know the details, but I'm pretty sure you can't put the blame of the Fukushima disaster on a single failure. e.g: - structure withstood magnitude 9 when designed for at least magnitude 8 - following the shock, rods went down successfully - power lines being down, generators powering water pumping started successfully - tsunami drowned oldest generators which were laid too low for a tsunami of that scale, backup batteries took charge - total amount of destruction around caused delays in grid power restoration so batteries eventually went down - core could not be cooled so it melted down, and fell to the containment vessel This is a tiny excerpt sequence of only parts of the events, but it can be seen that there were both multiple successes and multiple failures. Had there been a single failure, it would have stayed quite safe, but the cascade of events led to the disaster we know.

there 14y ago

Audi's strategy in the early 2000s for winning Le Mans 12- and 24-hour endurance races against Bentley was to engineer their cars around failure. The Bentley cars were faster, but Audi engineered theirs to be easier to fix. Over such long races, there were bound to be crashes or failures of various parts and the faster Audi could fix their cars, the quicker they could get back out on the track to outlast the faster cars. A great example of this was that Bentley used the lightest bolt they could everywhere on the car to minimize weight, but Audi used the same size bolt everywhere to minimize the number of tool changes needed to replace the parts.

WalterBright 14y ago

Airliners are designed to withstand failure, not to not have any failures. It's a change in perspective, and is very effective. Both the Fukishima plant and the Deep Water Horizon rig were unable to withstand single failures.

Mvandenbergh 14y ago
Nuclear power plants and oil rigs are also designed to withstand failure.
GoodIntentions 14y ago
>> Fukishima plant Actually it sounds like the plant failed due to some poor - in retrospect - design assumptions. The generators required to keep cooling systems running safely were buggered by the same disaster. "fail-over" - they got it half right.
- WalterBright 14y ago
  The problem is when the backup system is not independent of the primary, so one failure takes out the other. When I read about the step-by-step domino sequence of failures in the plant, the design was clearly deficient. The erroneous thinking was "the tsunami wall cannot fail". The correct thinking is "what happens when the tsunami wall is breached? what happens to the plant? how can the plant withstand that?" For example, the most obvious thing is to harden the backup diesel generators against flooding. This would not have been difficult nor expensive. Another solution would have been to have those generators located at some distance from the plant, so they could be repaired without irradiating the workers, and so a disaster at one would be less likely to affect the other. Another obviously poor design was to have the emergency hydrogen venting go into an enclosed space, where it can build up and explode, rather than venting it outside.
  - abduhl 14y ago
    I don't know what your background is but, based on your post, I am assuming it is not in civil or mechanical engineering. Please excuse me if I'm wrong. The design of this power plant probably went like most designs in civil engineering where extreme design loadings were defined (500-year quake, 100-year flood, etc.) and the plant was designed to meet the loads that would be imposed by the most critical event. For a critical piece of infrastructure these loading conditions are quite stringent. While these loadings drive the design at one end, budget drives the design at the other. Most civil projects tend to be just safe enough and not more. This leads to the realm of acceptable risk/failure. The tsunami wall failing under a 1000-year quake is an acceptable failure. The backup systems were most likely designed for progressively lower degrees of acceptable failure. At some point though it becomes too costly to over-design. Was hardening the backup generators against flooding "not difficult nor expensive"? I can't tell you that, I didn't design it. Based on the fact that they weren't hardened, I would say that it probably was difficult or expensive. Would placing the backup generators off site be a better idea than having them on site? Possibly. Possibly not. Every meter you move the generators away from the plant is another meter's worth of risk. How many redundant transmission lines do you need for a generator a mile away? How many redundant generators do you need now that you have the added risk of possibly losing a generator's transmission lines? Contrary to common belief, public works projects do not have an infinite budget. On top of this, working with municipalities generally results in ridiculous amounts of regulations regarding pricing, reviews, and permitting. While this maintains a minimum standard for designs, it also drives all designs down to that minimum standard. It is simply too costly to continually design for more and more unlikely events. Users of this site should be well familiar with this based on the number of Internet Explorer compatability comments I've read lately.
chriszf 14y ago
Actually, nuclear power plants are designed to withstand failure. Typically there are multiple backup plans to safeguard against failure of any one system. One comment I sometimes hear is, "They had to pump seawater in to cool things down? That's crazy! They should have had a backup plan." It turns out that nuclear plants are intentionally built near bodies of water because external cooling is a backup, should the primary cooling systems fail. Nothing, however, is designed to withstand systematic catastrophic failure of all of your backup plans.
lloeki 14y ago
> the Fukishima plant ... were unable to withstand single failures. I don't know the details, but I'm pretty sure you can't put the blame of the Fukushima disaster on a single failure. e.g: - structure withstood magnitude 9 when designed for at least magnitude 8 - following the shock, rods went down successfully - power lines being down, generators powering water pumping started successfully - tsunami drowned oldest generators which were laid too low for a tsunami of that scale, backup batteries took charge - total amount of destruction around caused delays in grid power restoration so batteries eventually went down - core could not be cooled so it melted down, and fell to the containment vessel This is a tiny excerpt sequence of only parts of the events, but it can be seen that there were both multiple successes and multiple failures. Had there been a single failure, it would have stayed quite safe, but the cascade of events led to the disaster we know.