3 ms·

Cycle times (the time it takes to process one wafer) can be in the range of a month. Any disruption therefore kills roughly a month (plus or minus) of output,

by dgacmu 7y ago

Cycle times (the time it takes to process one wafer) can be in the range of a month. Any disruption therefore kills roughly a month (plus or minus) of output, at least for wafers in certain steps. It's brutal. Fabs are engineered to have redundant power, but what's interesting is that the same thing happened to Samsung last year: https://www.anandtech.com/show/12535/power-outage-at-samsungs-fab-destroys-3-percent-of-global-nand-flash-output https://www.anandtech.com/show/12535/power-outage-at-samsung...

andruby 7y ago
This assumes all wafers in process at a given time are 100% waste after a power disruption. If true, that’s brutal indeed.
- emn13 7y ago
  Even if it weren't 100% wastage, it might take time and effort to find the salvagable bits. So when they restart, they'll might need to restart with an empty pipeline. By the time the salvagable bits are recovered it's conceivable the spots they could return to in the pipeline are filled; or merely that they're even slightly filled - enough for the risk to the just-starting production to discourage the addition. And even what they salvage might thus end up displacing new production - so while that might save them money, it might not increase throughput. And of course, it's just an estimate. Maybe the real damage will be different. I have absolutely no idea what I'm talking about, by the way ;-) - I'm just positing a hopefully plausible explanation why the ratio of wastage might not impact the amount of "lost" (in the sense of reduced from baseline) production, even if it help reduce the amount "lost" (in the sense of unrecoverable expended resources) production. I doubt non-experts can do much better than believe their own projections - assuming nobody with a real background here comes up with a solid reason why not.
igravious 7y ago
Interesting! That's my point though. If power outages hurt these fabs so severely why aren't their power supply systems more robust? I know it's easy for me to say but I'm having a hard time wrapping my head around it. Say in another engineering space where an hour of power outage means roughly an hour of downtime then you'd maybe not care so much. But if, as you link here, a 30 minute power outage can "destroy 3.5% of the global NAND supply for March" wouldn't they make sure they have 0 minutes of power outage – heck, that's nearly national security levels of threat – wouldn't the South Korean government install two (or three) sets of power lines from different parts of the grid. Or a local power source (diesel generators and a small coal power plant.) Expensive? Sure. But so is 3.5% of global NAND supply for a month?
- dgacmu 7y ago
  Power is honestly really hard -- for example, if you read through this list of datacenter power failure post-mortems: http://up2v.nl/2017/06/02/datacenter-complete-power-failures/ http://up2v.nl/2017/06/02/datacenter-complete-power-failures... there are a lot of individual failure cases. DC operators learn from each failure, but there are a lot of ways things can go wrong. Fabs can be upwards of 50MW, which puts them in the range of a good-sized datacenter, so the challenges probably end up similar. (I'm saying the last part carefully - I'm much more familiar with datacenter power design than fab power design!)

by dgacmu 7y ago

andruby 7y ago

This assumes all wafers in process at a given time are 100% waste after a power disruption. If true, that’s brutal indeed.

emn13 7y ago
Even if it weren't 100% wastage, it might take time and effort to find the salvagable bits. So when they restart, they'll might need to restart with an empty pipeline. By the time the salvagable bits are recovered it's conceivable the spots they could return to in the pipeline are filled; or merely that they're even slightly filled - enough for the risk to the just-starting production to discourage the addition. And even what they salvage might thus end up displacing new production - so while that might save them money, it might not increase throughput. And of course, it's just an estimate. Maybe the real damage will be different. I have absolutely no idea what I'm talking about, by the way ;-) - I'm just positing a hopefully plausible explanation why the ratio of wastage might not impact the amount of "lost" (in the sense of reduced from baseline) production, even if it help reduce the amount "lost" (in the sense of unrecoverable expended resources) production. I doubt non-experts can do much better than believe their own projections - assuming nobody with a real background here comes up with a solid reason why not.

igravious 7y ago

Interesting! That's my point though. If power outages hurt these fabs so severely why aren't their power supply systems more robust? I know it's easy for me to say but I'm having a hard time wrapping my head around it. Say in another engineering space where an hour of power outage means roughly an hour of downtime then you'd maybe not care so much. But if, as you link here, a 30 minute power outage can "destroy 3.5% of the global NAND supply for March" wouldn't they make sure they have 0 minutes of power outage – heck, that's nearly national security levels of threat – wouldn't the South Korean government install two (or three) sets of power lines from different parts of the grid. Or a local power source (diesel generators and a small coal power plant.) Expensive? Sure. But so is 3.5% of global NAND supply for a month?

dgacmu 7y ago
Power is honestly really hard -- for example, if you read through this list of datacenter power failure post-mortems: http://up2v.nl/2017/06/02/datacenter-complete-power-failures/ http://up2v.nl/2017/06/02/datacenter-complete-power-failures... there are a lot of individual failure cases. DC operators learn from each failure, but there are a lot of ways things can go wrong. Fabs can be upwards of 50MW, which puts them in the range of a good-sized datacenter, so the challenges probably end up similar. (I'm saying the last part carefully - I'm much more familiar with datacenter power design than fab power design!)