4 ms·

I wonder what failed in their redundant power supply because they surely have something. I hope the postmortem will be public !

by icefo 7y ago

igravious 7y ago
If my reading comprehension has not let me down then a 13 minute power disruption can cause them to lose 1/2 of their output for a quarter. Given the massive consequences of quite a short disruption maybe they need to figure out how to weather disruptions more robustly?
- iamgopal 7y ago
  Why individual process independently powered, if risk is this huge ? Why common power ?
  - igravious 7y ago
    Not quite understanding you, sorry.
- Tuna-Fish 7y ago
  > Given the massive consequences of quite a short disruption maybe they need to figure out how to weather disruptions more robustly? If you mean "they should not lose so much product when equipment loses power", that's just not possible. Modern semiconductor manufacturing involves hundreds of steps where the wafers need to soak in a chemical bath for a very specific time, and missing deadlines by a few seconds causes the entire wafer to fail. The question is very much: "why did their UPS fail?".
  - igravious 7y ago
    That is indeed what I was implying.
- dgacmu 7y ago
  Cycle times (the time it takes to process one wafer) can be in the range of a month. Any disruption therefore kills roughly a month (plus or minus) of output, at least for wafers in certain steps. It's brutal. Fabs are engineered to have redundant power, but what's interesting is that the same thing happened to Samsung last year: https://www.anandtech.com/show/12535/power-outage-at-samsungs-fab-destroys-3-percent-of-global-nand-flash-output https://www.anandtech.com/show/12535/power-outage-at-samsung...
  - andruby 7y ago
    This assumes all wafers in process at a given time are 100% waste after a power disruption. If true, that’s brutal indeed.
    - emn13 7y ago
      Even if it weren't 100% wastage, it might take time and effort to find the salvagable bits. So when they restart, they'll might need to restart with an empty pipeline. By the time the salvagable bits are recovered it's conceivable the spots they could return to in the pipeline are filled; or merely that they're even slightly filled - enough for the risk to the just-starting production to discourage the addition. And even what they salvage might thus end up displacing new production - so while that might save them money, it might not increase throughput. And of course, it's just an estimate. Maybe the real damage will be different. I have absolutely no idea what I'm talking about, by the way ;-) - I'm just positing a hopefully plausible explanation why the ratio of wastage might not impact the amount of "lost" (in the sense of reduced from baseline) production, even if it help reduce the amount "lost" (in the sense of unrecoverable expended resources) production. I doubt non-experts can do much better than believe their own projections - assuming nobody with a real background here comes up with a solid reason why not.
  - igravious 7y ago
    Interesting! That's my point though. If power outages hurt these fabs so severely why aren't their power supply systems more robust? I know it's easy for me to say but I'm having a hard time wrapping my head around it. Say in another engineering space where an hour of power outage means roughly an hour of downtime then you'd maybe not care so much. But if, as you link here, a 30 minute power outage can "destroy 3.5% of the global NAND supply for March" wouldn't they make sure they have 0 minutes of power outage – heck, that's nearly national security levels of threat – wouldn't the South Korean government install two (or three) sets of power lines from different parts of the grid. Or a local power source (diesel generators and a small coal power plant.) Expensive? Sure. But so is 3.5% of global NAND supply for a month?
- wyldfire 7y ago
  I can't imagine that something so business critical wouldn't have local power generation capacity to short-term sustain them through < 1 day outages (and rapid failover mechanisms). Either that mechanism failed somehow or there's more to this outage (maybe it damaged the production equipment as a result of spikes corresponding to the outage or repair).
  - igravious 7y ago
    If power supply issues hurt the business that severely then how come they allow it to happen is my point. I'm not saying I'm smarter than them, I'm not – I know there must be a reason, I can't figure it out.
perlgeek 7y ago
> I hope the postmortem will be public ! So do I. It turns out that backup power fails more often than one would hope. Generators fail to come online, batteries not performing as expected despite recent maintenance, switching gear failing, or the switching gear's safety mechanisms preventing a successful switch etc. Source: I work for a smallish ISP, and have heard lots of stories from the ISP community, and am always eager to read about outages when there's a public postmortem.
- firethief 7y ago
  It could be worse. At least you don't need that backup power to keep a nuclear reactor from doing anything unseemly
  - fest 7y ago
    I know, let's use the power from our own turbine as it spins down to power the emergency shutdown of our reactor! (see: Chernobyl)
tzs 7y ago
> I wonder what failed in their redundant power supply because they surely have something Based on comments on the site, it appears that even a very short power disruption can mess up semiconductor manufacturing. If that is true, then a backup power system that involved detecting an outage and starting up generators might be too slow. If based on generators, they'd either need to have the generators always running, or have a second redundant system based on batteries that can immediately take over during the time it takes to start the first redundant system. Or they could run their stuff off batteries all the time, with the batteries charged from the grid. They will still need something that can very quickly switch to the grid in the case of their own battery powered inverters failing. All of these are going to add complexity and cost that may drive up the effective cost of electricity enough that if may be cheaper in the long to simply go with the grid, if they are in a place with a reliable enough grid. Anyone know how reliable the grid is at their location?
- solarkraft 7y ago
  > Based on comments on the site, it appears that even a very short power disruption can mess up semiconductor manufacturing. > If that is true, then a backup power system that involved detecting an outage and starting up generators might be too slow. I expect them to have anticipated this. I expect them to have applied a system that would have worked for them, had it worked as designed. I would suspect the failure to have happened somewhere behind the redundant power supply (but then again why aren't the individual production steps not independently redundantly powered?).

I wonder what failed in their redundant power supply because they surely have something. I hope the postmortem will be public !

by icefo 7y ago

I wonder what failed in their redundant power supply because they surely have something. I hope the postmortem will be public !

igravious 7y ago

If my reading comprehension has not let me down then a 13 minute power disruption can cause them to lose 1/2 of their output for a quarter. Given the massive consequences of quite a short disruption maybe they need to figure out how to weather disruptions more robustly?

iamgopal 7y ago
Why individual process independently powered, if risk is this huge ? Why common power ?
- igravious 7y ago
  Not quite understanding you, sorry.
Tuna-Fish 7y ago
> Given the massive consequences of quite a short disruption maybe they need to figure out how to weather disruptions more robustly? If you mean "they should not lose so much product when equipment loses power", that's just not possible. Modern semiconductor manufacturing involves hundreds of steps where the wafers need to soak in a chemical bath for a very specific time, and missing deadlines by a few seconds causes the entire wafer to fail. The question is very much: "why did their UPS fail?".
- igravious 7y ago
  That is indeed what I was implying.
dgacmu 7y ago
Cycle times (the time it takes to process one wafer) can be in the range of a month. Any disruption therefore kills roughly a month (plus or minus) of output, at least for wafers in certain steps. It's brutal. Fabs are engineered to have redundant power, but what's interesting is that the same thing happened to Samsung last year: https://www.anandtech.com/show/12535/power-outage-at-samsungs-fab-destroys-3-percent-of-global-nand-flash-output https://www.anandtech.com/show/12535/power-outage-at-samsung...
- andruby 7y ago
  This assumes all wafers in process at a given time are 100% waste after a power disruption. If true, that’s brutal indeed.
  - emn13 7y ago
    Even if it weren't 100% wastage, it might take time and effort to find the salvagable bits. So when they restart, they'll might need to restart with an empty pipeline. By the time the salvagable bits are recovered it's conceivable the spots they could return to in the pipeline are filled; or merely that they're even slightly filled - enough for the risk to the just-starting production to discourage the addition. And even what they salvage might thus end up displacing new production - so while that might save them money, it might not increase throughput. And of course, it's just an estimate. Maybe the real damage will be different. I have absolutely no idea what I'm talking about, by the way ;-) - I'm just positing a hopefully plausible explanation why the ratio of wastage might not impact the amount of "lost" (in the sense of reduced from baseline) production, even if it help reduce the amount "lost" (in the sense of unrecoverable expended resources) production. I doubt non-experts can do much better than believe their own projections - assuming nobody with a real background here comes up with a solid reason why not.
- igravious 7y ago
  Interesting! That's my point though. If power outages hurt these fabs so severely why aren't their power supply systems more robust? I know it's easy for me to say but I'm having a hard time wrapping my head around it. Say in another engineering space where an hour of power outage means roughly an hour of downtime then you'd maybe not care so much. But if, as you link here, a 30 minute power outage can "destroy 3.5% of the global NAND supply for March" wouldn't they make sure they have 0 minutes of power outage – heck, that's nearly national security levels of threat – wouldn't the South Korean government install two (or three) sets of power lines from different parts of the grid. Or a local power source (diesel generators and a small coal power plant.) Expensive? Sure. But so is 3.5% of global NAND supply for a month?
wyldfire 7y ago
I can't imagine that something so business critical wouldn't have local power generation capacity to short-term sustain them through < 1 day outages (and rapid failover mechanisms). Either that mechanism failed somehow or there's more to this outage (maybe it damaged the production equipment as a result of spikes corresponding to the outage or repair).
- igravious 7y ago
  If power supply issues hurt the business that severely then how come they allow it to happen is my point. I'm not saying I'm smarter than them, I'm not – I know there must be a reason, I can't figure it out.

perlgeek 7y ago

> I hope the postmortem will be public ! So do I. It turns out that backup power fails more often than one would hope. Generators fail to come online, batteries not performing as expected despite recent maintenance, switching gear failing, or the switching gear's safety mechanisms preventing a successful switch etc. Source: I work for a smallish ISP, and have heard lots of stories from the ISP community, and am always eager to read about outages when there's a public postmortem.

firethief 7y ago
It could be worse. At least you don't need that backup power to keep a nuclear reactor from doing anything unseemly
- fest 7y ago
  I know, let's use the power from our own turbine as it spins down to power the emergency shutdown of our reactor! (see: Chernobyl)

tzs 7y ago

> I wonder what failed in their redundant power supply because they surely have something Based on comments on the site, it appears that even a very short power disruption can mess up semiconductor manufacturing. If that is true, then a backup power system that involved detecting an outage and starting up generators might be too slow. If based on generators, they'd either need to have the generators always running, or have a second redundant system based on batteries that can immediately take over during the time it takes to start the first redundant system. Or they could run their stuff off batteries all the time, with the batteries charged from the grid. They will still need something that can very quickly switch to the grid in the case of their own battery powered inverters failing. All of these are going to add complexity and cost that may drive up the effective cost of electricity enough that if may be cheaper in the long to simply go with the grid, if they are in a place with a reliable enough grid. Anyone know how reliable the grid is at their location?

solarkraft 7y ago
> Based on comments on the site, it appears that even a very short power disruption can mess up semiconductor manufacturing. > If that is true, then a backup power system that involved detecting an outage and starting up generators might be too slow. I expect them to have anticipated this. I expect them to have applied a system that would have worked for them, had it worked as designed. I would suspect the failure to have happened somewhere behind the redundant power supply (but then again why aren't the individual production steps not independently redundantly powered?).