8 ms·

TSMC Arizona outage saw fab halt, Apple wafers scrapped

220 points by speckx 1y ago

taurath 1y ago
In September
- joecool1029 1y ago
  But first being reported now: It was only speculation on the financial reports before this. How quickly do they normally report disruptions like this? I wouldn't think it would have to be too quickly since I've heard about fab disruptions from fires and such since the early 2000's. Probably just sometime after quarterly reporting to set the record straight? Why not in the report?
  - samus 1y ago
    I also had the impression from the report that shareholders were miffed about this Q3 snag, so they had to publish this even though they were about to treat this as business as usual.
bob1029 1y ago
> forcing the facility to shut down for at least a few hours > As a result, the company had to scrap thousands of wafers Anything involving wet chemistry, photoresist, furnaces, etc. is very time-constrained. You can't let wafers sit around indefinitely. Certain process steps must be followed up very quickly to avoid scrap. This is why you dont see redundant power for manufacturing lines. A 3nm line needs hundreds of megawatts to operate. You cant clear queued lots without a fully functional line. There's not much you could save by keeping part of the line operational.
- tantalor 1y ago
  Good idea for a Factorio mod? A new failure mode resets output progress back to zero if you lose power or some other input while crafting. You could design circuit networks to cut power to non-essential systems so the rest of the factory can keep producing.
  - ActorNightly 1y ago
    Someone needs to make the whole chip manufacturing process into a factorio like game and let the gamers optimize it, then build the factories around that.
    - gnatman 1y ago
      Like Ender’s Game but instead of intergalactic shooting war it’s international chip war.
  - sidewndr46 1y ago
    Isn't that what spoilage is?
    - helpfulclippy 1y ago
      I thought of spoilage as a mechanic that punishes overproduction.
      - blmarket 1y ago
        It's a constraint to process item within limited time (regardless of overproduction or power outage). Matching with the problem description. Surely the reality might be much more complex (like... the yield/quality drop by time function?)
  - tetha 1y ago
    Some mods in modded minecraft had that and it's a very punishing mechanic unless implemented well. It eats all of your power and usually also very expensive items very quickly usually. Assume you have like 600RF/tick generated, common with certain generator constellations. 1 tick - 1024 RF and one input consumed, crafting fails due to not enough power. 1 tick wait, 1 tick, 1024 RF and one input consumed, ... This can void 10+ items / second, which can hurt very badly. Even for common items in fact. It also tends to kick you while you're down, because it only kicks in if everything else is already failing. Then the only thing to continue functioning is the thing voiding your energy and your expensive items. Or even worse, if you did one miscalculation about your power grid, and then all of your resources are gone, often before you can react. It can be interesting in the right packs, but it is Gregtech level hardness.
- Kye 1y ago
  A video showing those steps, for the curious: https://www.youtube.com/watch?v=dX9CGRZwD-w https://www.youtube.com/watch?v=dX9CGRZwD-w It's probably not 100% identical to TSMC's process.
- sevensor 1y ago
  It didn’t happen, but the facilities team at the fab where I worked was seriously considering installing a flywheel to cover power bumps. What I don’t get about this story is how this actually happened. All our process gasses were out in a tank farm and we knew how much pressure we had. We would have stopped the line if there wasn’t enough to proceed. Were they separating air onsite or something?
  - bobmcnamara 1y ago
    Yes, Linde has an onsite plant and is building two more. For some processes, stopping will botch the wafer. In the event of a gas shortage, do plants plan which lines to take down first, and which lines should complete a process step?
    - sevensor 1y ago
      The way this worked at the fab where I was, was that facilities would have paged everybody, and whoever needed to hold wafers would do so. You could mark your equipment down or unavailable for a particular step. I don’t know what we would have done if it was “hey, we lost dry nitrogen a minute ago.” I think at that point you lose a lot of wafers in wet cleans. In the case of a power interruption at the fab, consequences were highly dependent on the equipment and the unit process. A prolonged power interruption to diffusion was the worst case scenario. You’d have 150 wafers in the furnace, and any significant deviation from the nominal temperature profile meant they were all scrap. Worse, if the furnace cooled off, you had to scrap the quartz boat the wafers rode in, too. Other processes had a smaller blast radius but were even more of a headache to disposition. Implant, you’d lose beam and probably lose vacuum too. Then the wafer in the chamber would be dusted and in an indeterminate state, and the rest of the wafers you’d have to sleuth out whether they were implanted or not. Sometimes you’d have a lot sitting in the end station and it wouldn’t be clear whether or not it had been run at all. At least in photolithography you could tell whether or not a wafer was patterned by looking at it.
  - jaggederest 1y ago
    I was very impressed by the modest little fab I worked at having thousands of lead acid batteries for momentary takeover, and 8 five-megawatt locomotive engines for longer term redundancy. Apparently their steady state usage was 25MW, which allowed still having a hot spare and concurrent downtime for two of the locomotive generator units.
- j_walter 1y ago
  TSMC has backup generators in their AZ fab. You actually have to have backup power or a few hundred millisecond blip could cause days or weeks of tool down time. You should see what happens when you lose the ability to keep a clean room at temp/humidity/airflow...it's weeks or months.
angelgonzales 1y ago
This isn’t very big news. Issues occur during bring-up often. Linde’s processes are possibly so power intensive that failing over to generator power is not possible. TSMC is right to put Linde on notice since Linde should have a PFMEA and control plan to eliminate any root causes for downtime. I suspect in the long term TSMC has plans to insource this if the issue persists. Scrap happens sometimes during manufacturing, if the writer only has journalism experience and no manufacturing experience then they may not have a conceptual understanding of acceptable first pass yield. After all, the TSMC logo features failing parts!
- FaradayRotation 1y ago
  In many ways I agree with you, but the problem statement (constrained/exhausted gas supply from vendor) makes it seems like this was not just line down, but the whole factory stopped for a few hours. Line down is a miserable migrane but still managable... while a whole factory stoppage makes a lobotomy seem like a good idea. It also sounds like there was not enough forewarning to park critical customer wafers in a "safe" stage of the process. Even so, I also would still call this another monday at a semiconductor factory. Welcome! Here we play a nearly endless game of whack-a-mole. Here's your mallet and your towel. Now whack enough of the moles hard enough until they stop coming back (at least through the same holes). Beware the alpha moles. By any road, I am surprised to see even this high-level perspective on a quality event disclosed to the mainstream public; I thought this was not standard practice. I enjoyed the read.
  - itopaloglu83 1y ago
    Just curious, would a full factory stoppage require recalibration or revalidation of certain equipment? Or is it more like an atmospheric issue that only affects the product.
    - franktankbank 1y ago
      I'm also curious. Its not like the power went out and machines unsafely shutdown though.
agentifysh 1y ago
seems like what is often downplayed or silent on American media is the cultural mismatch between TSMC taiwanese engineers and their american counterparts so it always comes to those out of the loop as a bit of a surprise but from what I've read from individual Taiwanese workers and their feedback its clear that there is significant regret from one side. and it doesn't seem to limited to just TSMC but another large company as of recent that receive icey reception for their large investment in America manufacturing. i think this is a big reason why lot of these jobs simply wouldn't stay in america as the consumer would not be able to foot the costs added by "cultural premium" faster than what innovation can reduce.
- itake 1y ago
  Perhaps if the US workers earned OT like TW workers do, the "culture" gap would shrink.
  - limagnolia 1y ago
    Is OT "overtime"? How is it legal not to pay overtime in any US factory? Unless they are salaried (exempt)?
    - TimorousBestie 1y ago
      A lot of the workers there probably are exempt under American law. I’m not an expert on Taiwanese labor laws but their list of exempt labor categories in the LSA is much shorter than the one in the American FLSA.
    - itake 1y ago
      yeah, overtime. My cousin is an engineer at TSMC (who worked both in Tainin and now in Arizona) and is w-2 exempt.
  - j_walter 1y ago
    TW workers have a majority of their compensation in bonuses, so the OT portion is quite small and many do not even bother to ask for it. The overall compensation between a TW and US engineer at TSMC is also significant. Not to mention the lowest paid hourly workers...where in TW they make 2-3X minimum wage, but in the US it's like 1.25X.
behnamoh 1y ago
Does it further delay the launch of M5 Max/Ultra?
deleted 1y ago
[deleted]
richisferezs 1y ago
guys is it me or sonnet 4.5 just became like 10x worst ?
perihelions 1y ago
Some HN threads about the past incidents mentioned in OP, https://news.ycombinator.com/item?id=17686310 https://news.ycombinator.com/item?id=17686310 ("Computer Virus Cripples Several Taiwan Semiconductor Plants (bloomberg.com)"—2018, 100 comments) https://news.ycombinator.com/item?id=19214952 https://news.ycombinator.com/item?id=19214952 ("TSMC's Photoresist Material Incident: $550M Loss (anandtech.com)"—2019, 15 comments)
Animats 1y ago
This was apparently a Linde installation custom built for TSMC in Arizona.[1] Nitrogen, oxygen, and argon are extracted from air on-site and purified. That's Linde's primary business; liquefying and distilling air. This isn't some little local company or a company operating outside their area of expertise. Those gases are storeable, so it's surprising there wasn't enough tank capacity to deal with outages. The site plan [2] shows "Gas Plant 1", and future "Gas Plant 2" and "Gas Plant 3". The gas plants are across a small road from the fab and feed the plant directly. Once Gas Plants 2 and 3 were built, there would be redundancy, but at this stage, there isn't a backup. The plan doesn't show a large tank farm, so they can't store gases in bulk. [1] https://www.aztechcouncil.org/utility-company-makes-progress-on-infrastructure-for-taiwan-semiconductor-project-in-north-phoenix/ https://www.aztechcouncil.org/utility-company-makes-progress... [2] https://semiwiki.com/forum/threads/tsmc-phoenix-arizona-fab-site-plan.16066/ https://semiwiki.com/forum/threads/tsmc-phoenix-arizona-fab-...
- NoiseBert69 1y ago
  Linde is huge. They can produce and offer all gases in all available purity classes.
  - deleted 1y ago
    [deleted]
  - jack_tripper 1y ago
    Is this an ad?
    - RealityVoid 1y ago
      I don't think Linde™ needs an add, everyone knows Linde™ is the most reliable partner in producing and storing gases in all available purity classes. (joke off, it's probably not an add, but they were excited to share the reason you see Linde on all sorts of gas tanks all over the place. It's actually quite common and if you see it once you see it everywhere. )
caycep 1y ago
the rebels have hit the tibonna gas supply I see
jason-richar15 1y ago
[dead]
therealfiona 1y ago
Linde shut down the shop that made these machines. So I'm sure they're shaking in their boots about being put in notice...

220 points by speckx 1y ago

taurath 1y ago

In September

joecool1029 1y ago
But first being reported now: It was only speculation on the financial reports before this. How quickly do they normally report disruptions like this? I wouldn't think it would have to be too quickly since I've heard about fab disruptions from fires and such since the early 2000's. Probably just sometime after quarterly reporting to set the record straight? Why not in the report?
- samus 1y ago
  I also had the impression from the report that shareholders were miffed about this Q3 snag, so they had to publish this even though they were about to treat this as business as usual.

bob1029 1y ago

> forcing the facility to shut down for at least a few hours > As a result, the company had to scrap thousands of wafers Anything involving wet chemistry, photoresist, furnaces, etc. is very time-constrained. You can't let wafers sit around indefinitely. Certain process steps must be followed up very quickly to avoid scrap. This is why you dont see redundant power for manufacturing lines. A 3nm line needs hundreds of megawatts to operate. You cant clear queued lots without a fully functional line. There's not much you could save by keeping part of the line operational.

tantalor 1y ago
Good idea for a Factorio mod? A new failure mode resets output progress back to zero if you lose power or some other input while crafting. You could design circuit networks to cut power to non-essential systems so the rest of the factory can keep producing.
- ActorNightly 1y ago
  Someone needs to make the whole chip manufacturing process into a factorio like game and let the gamers optimize it, then build the factories around that.
  - gnatman 1y ago
    Like Ender’s Game but instead of intergalactic shooting war it’s international chip war.
- sidewndr46 1y ago
  Isn't that what spoilage is?
  - helpfulclippy 1y ago
    I thought of spoilage as a mechanic that punishes overproduction.
    - blmarket 1y ago
      It's a constraint to process item within limited time (regardless of overproduction or power outage). Matching with the problem description. Surely the reality might be much more complex (like... the yield/quality drop by time function?)
- tetha 1y ago
  Some mods in modded minecraft had that and it's a very punishing mechanic unless implemented well. It eats all of your power and usually also very expensive items very quickly usually. Assume you have like 600RF/tick generated, common with certain generator constellations. 1 tick - 1024 RF and one input consumed, crafting fails due to not enough power. 1 tick wait, 1 tick, 1024 RF and one input consumed, ... This can void 10+ items / second, which can hurt very badly. Even for common items in fact. It also tends to kick you while you're down, because it only kicks in if everything else is already failing. Then the only thing to continue functioning is the thing voiding your energy and your expensive items. Or even worse, if you did one miscalculation about your power grid, and then all of your resources are gone, often before you can react. It can be interesting in the right packs, but it is Gregtech level hardness.
Kye 1y ago
A video showing those steps, for the curious: https://www.youtube.com/watch?v=dX9CGRZwD-w https://www.youtube.com/watch?v=dX9CGRZwD-w It's probably not 100% identical to TSMC's process.
sevensor 1y ago
It didn’t happen, but the facilities team at the fab where I worked was seriously considering installing a flywheel to cover power bumps. What I don’t get about this story is how this actually happened. All our process gasses were out in a tank farm and we knew how much pressure we had. We would have stopped the line if there wasn’t enough to proceed. Were they separating air onsite or something?
- bobmcnamara 1y ago
  Yes, Linde has an onsite plant and is building two more. For some processes, stopping will botch the wafer. In the event of a gas shortage, do plants plan which lines to take down first, and which lines should complete a process step?
  - sevensor 1y ago
    The way this worked at the fab where I was, was that facilities would have paged everybody, and whoever needed to hold wafers would do so. You could mark your equipment down or unavailable for a particular step. I don’t know what we would have done if it was “hey, we lost dry nitrogen a minute ago.” I think at that point you lose a lot of wafers in wet cleans. In the case of a power interruption at the fab, consequences were highly dependent on the equipment and the unit process. A prolonged power interruption to diffusion was the worst case scenario. You’d have 150 wafers in the furnace, and any significant deviation from the nominal temperature profile meant they were all scrap. Worse, if the furnace cooled off, you had to scrap the quartz boat the wafers rode in, too. Other processes had a smaller blast radius but were even more of a headache to disposition. Implant, you’d lose beam and probably lose vacuum too. Then the wafer in the chamber would be dusted and in an indeterminate state, and the rest of the wafers you’d have to sleuth out whether they were implanted or not. Sometimes you’d have a lot sitting in the end station and it wouldn’t be clear whether or not it had been run at all. At least in photolithography you could tell whether or not a wafer was patterned by looking at it.
- jaggederest 1y ago
  I was very impressed by the modest little fab I worked at having thousands of lead acid batteries for momentary takeover, and 8 five-megawatt locomotive engines for longer term redundancy. Apparently their steady state usage was 25MW, which allowed still having a hot spare and concurrent downtime for two of the locomotive generator units.
j_walter 1y ago
TSMC has backup generators in their AZ fab. You actually have to have backup power or a few hundred millisecond blip could cause days or weeks of tool down time. You should see what happens when you lose the ability to keep a clean room at temp/humidity/airflow...it's weeks or months.

angelgonzales 1y ago

This isn’t very big news. Issues occur during bring-up often. Linde’s processes are possibly so power intensive that failing over to generator power is not possible. TSMC is right to put Linde on notice since Linde should have a PFMEA and control plan to eliminate any root causes for downtime. I suspect in the long term TSMC has plans to insource this if the issue persists. Scrap happens sometimes during manufacturing, if the writer only has journalism experience and no manufacturing experience then they may not have a conceptual understanding of acceptable first pass yield. After all, the TSMC logo features failing parts!

FaradayRotation 1y ago
In many ways I agree with you, but the problem statement (constrained/exhausted gas supply from vendor) makes it seems like this was not just line down, but the whole factory stopped for a few hours. Line down is a miserable migrane but still managable... while a whole factory stoppage makes a lobotomy seem like a good idea. It also sounds like there was not enough forewarning to park critical customer wafers in a "safe" stage of the process. Even so, I also would still call this another monday at a semiconductor factory. Welcome! Here we play a nearly endless game of whack-a-mole. Here's your mallet and your towel. Now whack enough of the moles hard enough until they stop coming back (at least through the same holes). Beware the alpha moles. By any road, I am surprised to see even this high-level perspective on a quality event disclosed to the mainstream public; I thought this was not standard practice. I enjoyed the read.
- itopaloglu83 1y ago
  Just curious, would a full factory stoppage require recalibration or revalidation of certain equipment? Or is it more like an atmospheric issue that only affects the product.
  - franktankbank 1y ago
    I'm also curious. Its not like the power went out and machines unsafely shutdown though.

agentifysh 1y ago

seems like what is often downplayed or silent on American media is the cultural mismatch between TSMC taiwanese engineers and their american counterparts so it always comes to those out of the loop as a bit of a surprise but from what I've read from individual Taiwanese workers and their feedback its clear that there is significant regret from one side. and it doesn't seem to limited to just TSMC but another large company as of recent that receive icey reception for their large investment in America manufacturing. i think this is a big reason why lot of these jobs simply wouldn't stay in america as the consumer would not be able to foot the costs added by "cultural premium" faster than what innovation can reduce.

itake 1y ago
Perhaps if the US workers earned OT like TW workers do, the "culture" gap would shrink.
- limagnolia 1y ago
  Is OT "overtime"? How is it legal not to pay overtime in any US factory? Unless they are salaried (exempt)?
  - TimorousBestie 1y ago
    A lot of the workers there probably are exempt under American law. I’m not an expert on Taiwanese labor laws but their list of exempt labor categories in the LSA is much shorter than the one in the American FLSA.
  - itake 1y ago
    yeah, overtime. My cousin is an engineer at TSMC (who worked both in Tainin and now in Arizona) and is w-2 exempt.
- j_walter 1y ago
  TW workers have a majority of their compensation in bonuses, so the OT portion is quite small and many do not even bother to ask for it. The overall compensation between a TW and US engineer at TSMC is also significant. Not to mention the lowest paid hourly workers...where in TW they make 2-3X minimum wage, but in the US it's like 1.25X.

behnamoh 1y ago

Does it further delay the launch of M5 Max/Ultra?

deleted 1y ago

[deleted]

richisferezs 1y ago

guys is it me or sonnet 4.5 just became like 10x worst ?

perihelions 1y ago

Some HN threads about the past incidents mentioned in OP, https://news.ycombinator.com/item?id=17686310 https://news.ycombinator.com/item?id=17686310 ("Computer Virus Cripples Several Taiwan Semiconductor Plants (bloomberg.com)"—2018, 100 comments) https://news.ycombinator.com/item?id=19214952 https://news.ycombinator.com/item?id=19214952 ("TSMC's Photoresist Material Incident: $550M Loss (anandtech.com)"—2019, 15 comments)

Animats 1y ago

This was apparently a Linde installation custom built for TSMC in Arizona.[1] Nitrogen, oxygen, and argon are extracted from air on-site and purified. That's Linde's primary business; liquefying and distilling air. This isn't some little local company or a company operating outside their area of expertise. Those gases are storeable, so it's surprising there wasn't enough tank capacity to deal with outages. The site plan [2] shows "Gas Plant 1", and future "Gas Plant 2" and "Gas Plant 3". The gas plants are across a small road from the fab and feed the plant directly. Once Gas Plants 2 and 3 were built, there would be redundancy, but at this stage, there isn't a backup. The plan doesn't show a large tank farm, so they can't store gases in bulk. [1] https://www.aztechcouncil.org/utility-company-makes-progress-on-infrastructure-for-taiwan-semiconductor-project-in-north-phoenix/ https://www.aztechcouncil.org/utility-company-makes-progress... [2] https://semiwiki.com/forum/threads/tsmc-phoenix-arizona-fab-site-plan.16066/ https://semiwiki.com/forum/threads/tsmc-phoenix-arizona-fab-...

NoiseBert69 1y ago
Linde is huge. They can produce and offer all gases in all available purity classes.
- deleted 1y ago
  [deleted]
- jack_tripper 1y ago
  Is this an ad?
  - RealityVoid 1y ago
    I don't think Linde™ needs an add, everyone knows Linde™ is the most reliable partner in producing and storing gases in all available purity classes. (joke off, it's probably not an add, but they were excited to share the reason you see Linde on all sorts of gas tanks all over the place. It's actually quite common and if you see it once you see it everywhere. )

caycep 1y ago

the rebels have hit the tibonna gas supply I see

jason-richar15 1y ago

[dead]

therealfiona 1y ago

Linde shut down the shop that made these machines. So I'm sure they're shaking in their boots about being put in notice...