6 ms·

I would be most grateful if someone could please explain what sort of tools are likely to be used here, and why a power loss to those tools would ruin days/week

I would be most grateful if someone could please explain what sort of tools are likely to be used here, and why a power loss to those tools would ruin days/weeks/months worth of output relative to the time they were offline?

kop316 7y ago
I imagine that the process is very highly pipelined and optimized, and I would imagine that they had some sort of backup (generator) that failed. One analogy is to think about if you had a batch script that you were working on that touches a lot of files (1000s). Now imagine power was cut and the batch script was interupted because the computer turned off, but that computer hasnt been turned off in a long time (say it was a server). First, you have to turn the server back on after a power outage. Was there corrupted files in it? You have to now get that server in a known working state, and if you have kept it on for years....then you may be in a world of hurt. NOw you got your server up and running. You have the option of going through each of the 1000s of files your script was working on....but that will take time. Does it make sense to start from scratch? You will have to through our all the files you were working on, but at least you can start that script again. You could attempt to salvage every file, but that will also take time too.
- paranoidrobot 7y ago
  Another analogy is that you have a bakery that produces soufflés on an industrial scale. These soufflés take two months to bake. The baking has to be done in such a precise fashion that even fractions of a degree in variance results in the entire batch being ruined. Worse still, it takes a long time to bring the oven up to temperature and stabilise it on the precise temperature. You can't just scrap the batch that's ruined and start production again.
  - watsocd 7y ago
    I like this analogy. Lets expand. These soufflés take two months to bake but you need soufflés every day for sales. What does this mean? You always have 2 months of soufflés in various states of production at all times. Now you lose power and all these as very fragile soufflés in production are lost because of the power failure. Furthermore, it will take you two months to get the first soufflés off the restarted production line.
furi 7y ago
Semiconductor manufacturing involves a lot of precisely controlled processes. You put the wafers into furnaces and pass a gas over them for X time and Y flow rate at Z temperature to impregnate the wafer with various chemicals. You put them in low pressure plasma environments to etch them, again for X time at Y flow rate. There are half a dozen more of these as well, like applying metal and implanting ions. These values are experimentally tightened to get the highest possible accuracy to the desired effect and improve the number of working chips that leave the factory. If the power cuts out you don't know what conditions the wafer experienced while the system was winding down completely uncontrolled and your processes haven't been designed for the wafer going through the ramp up twice. The reason why it's lost so much output is because modern semiconductor processes have hundreds of steps and (I believe) a lead time in the months, so the amount of material that's in flight at any one instant has to be huge to get any reasonable throughput.
- agumonkey 7y ago
  So they realized late that an early step of the pipeline was off, rendering everything that has been through flawed ?
  - BeeOnRope 7y ago
    More like many steps in parallel would all be affected by a power outage, and much or all of the wafers in progress were ruined or simply not economical to recover.
  - CydeWeys 7y ago
    No, they experienced a power outage which has ruined every chip that was currently at any point in the pipeline. The realization would be immediately (as soon as the power goes out).
    - shaklee3 7y ago
      It said that it was 1/2 their production output for one quarter. Is that really possible for an instantaneous power outage?
TheAceOfHearts 7y ago
Yesterday this issue was being discussed in /r/DataHoarder and I asked a similar question: https://www.reddit.com/r/DataHoarder/comments/c6mt9l/a_13_minute_power_loss_cost_wd_6eb_worth_of/esaqok5/?context=3 https://www.reddit.com/r/DataHoarder/comments/c6mt9l/a_13_mi...
AFascistWorld 7y ago
Acids, the manufacturing basically is controlling the etching process to get rid of unwanted parts and keep the designed metal circuits. When you lost power, you are not sure if the chips stayed with acids for too long or too short, or coated with unwanted amount of materials, the uncertainty kills the yield rate, which can be already low since memory chips require repeated stacking nowadays. Similar to https://i.stack.imgur.com/yTQqw.jpg https://i.stack.imgur.com/yTQqw.jpg

kop316 7y ago

I imagine that the process is very highly pipelined and optimized, and I would imagine that they had some sort of backup (generator) that failed. One analogy is to think about if you had a batch script that you were working on that touches a lot of files (1000s). Now imagine power was cut and the batch script was interupted because the computer turned off, but that computer hasnt been turned off in a long time (say it was a server). First, you have to turn the server back on after a power outage. Was there corrupted files in it? You have to now get that server in a known working state, and if you have kept it on for years....then you may be in a world of hurt. NOw you got your server up and running. You have the option of going through each of the 1000s of files your script was working on....but that will take time. Does it make sense to start from scratch? You will have to through our all the files you were working on, but at least you can start that script again. You could attempt to salvage every file, but that will also take time too.

paranoidrobot 7y ago
Another analogy is that you have a bakery that produces soufflés on an industrial scale. These soufflés take two months to bake. The baking has to be done in such a precise fashion that even fractions of a degree in variance results in the entire batch being ruined. Worse still, it takes a long time to bring the oven up to temperature and stabilise it on the precise temperature. You can't just scrap the batch that's ruined and start production again.
- watsocd 7y ago
  I like this analogy. Lets expand. These soufflés take two months to bake but you need soufflés every day for sales. What does this mean? You always have 2 months of soufflés in various states of production at all times. Now you lose power and all these as very fragile soufflés in production are lost because of the power failure. Furthermore, it will take you two months to get the first soufflés off the restarted production line.

furi 7y ago

Semiconductor manufacturing involves a lot of precisely controlled processes. You put the wafers into furnaces and pass a gas over them for X time and Y flow rate at Z temperature to impregnate the wafer with various chemicals. You put them in low pressure plasma environments to etch them, again for X time at Y flow rate. There are half a dozen more of these as well, like applying metal and implanting ions. These values are experimentally tightened to get the highest possible accuracy to the desired effect and improve the number of working chips that leave the factory. If the power cuts out you don't know what conditions the wafer experienced while the system was winding down completely uncontrolled and your processes haven't been designed for the wafer going through the ramp up twice. The reason why it's lost so much output is because modern semiconductor processes have hundreds of steps and (I believe) a lead time in the months, so the amount of material that's in flight at any one instant has to be huge to get any reasonable throughput.

agumonkey 7y ago
So they realized late that an early step of the pipeline was off, rendering everything that has been through flawed ?
- BeeOnRope 7y ago
  More like many steps in parallel would all be affected by a power outage, and much or all of the wafers in progress were ruined or simply not economical to recover.
- CydeWeys 7y ago
  No, they experienced a power outage which has ruined every chip that was currently at any point in the pipeline. The realization would be immediately (as soon as the power goes out).
  - shaklee3 7y ago
    It said that it was 1/2 their production output for one quarter. Is that really possible for an instantaneous power outage?

TheAceOfHearts 7y ago

Yesterday this issue was being discussed in /r/DataHoarder and I asked a similar question: https://www.reddit.com/r/DataHoarder/comments/c6mt9l/a_13_minute_power_loss_cost_wd_6eb_worth_of/esaqok5/?context=3 https://www.reddit.com/r/DataHoarder/comments/c6mt9l/a_13_mi...

AFascistWorld 7y ago

Acids, the manufacturing basically is controlling the etching process to get rid of unwanted parts and keep the designed metal circuits. When you lost power, you are not sure if the chips stayed with acids for too long or too short, or coated with unwanted amount of materials, the uncertainty kills the yield rate, which can be already low since memory chips require repeated stacking nowadays. Similar to https://i.stack.imgur.com/yTQqw.jpg https://i.stack.imgur.com/yTQqw.jpg