27 ms·

I interviewed a guy from the NOAA to ask about that. The gist is 1) the military has hardened satellites 2) they have a phone-tree system for alerting grid oper

I interviewed a guy from the NOAA to ask about that. The gist is 1) the military has hardened satellites 2) they have a phone-tree system for alerting grid operators and 3) many high voltage transformers have back-up systems in hardened storage. [1] https://ryanblakeley.net/p/solar-flare https://ryanblakeley.net/p/solar-flare

idiotsecant 3y ago
I won't comment on 1 and 2 but 3) is laughably untrue. I guess the word 'many' might be stretched to mean 'a few' but I've worked at more than one very large utility directly as an engineer responsible for design of large generation and transmission interface systems and these transformers are not only not spare, they're almost impossible to get quickly even at current market demand. If you toasted even 10 percent of the large generation and transmission transformers in North America it would be years before the grid would even come close to working again, and that's if there was nation state level resources applied. These are specialized machines that require specialized labor and specialized materials from a specialized international supply chain to construct. I'd make the bet that total recovery would be decades.
- foobarian 3y ago
  This keeps coming up and I keep wondering how in the world are large transformers that complicated to build. And I don't mean that in a dismissive way but a "boy do I have a gap in my knowledge of the world" way :-)
  - s1artibartfast 3y ago
    I think that is a pretty common reaction when it comes to physical/mechanical objects. Another way of thinking about it which might be helpful is. Hundreds or thousands of very talented engineers have been working non-stop for 150+ years in a transformer arms race. This has resulted in incredibly high performance designs and incredibly high requirements.
    - foobarian 3y ago
      No doubt. But same could be said of, say, automobiles, or microchips, and many other things. Even larger objects like power plant turbines seem relatively frequently manufactured. Large transformers on the other hand appear to be closer to nuclear power plants in rarity/difficulty of replacement, and as a result for me they have a very high difficulty-of-replacement-to-perceived-complexity ratio.
      - idiotsecant 3y ago
        Large transformers are built to a very specific custom specification. Each one is a custom engineered unit. A transformer is basically a sealed vessel full of volatile oil that you pass an extremely large voltage and current through. The machine has to be perfect. The smallest clearance issue, insulation problem, etc and you have a bomb. And that's just making it safe. We haven't even talked about making it effective. Transformers are the sort of think that are not rocket science to build, but every single detail is very very important. It requires a team of people to design, construct, and validate and all these people need to have the skills from doing it many times before. These people and facilities simply don't exist in large enough numbers because they are very expensive. There are also supply chain issues to consider - the steel used in transformer magnetic circuits for example is a very specific metallurgy and construction. During COVID it was quite difficult to find. Poor steel means poor efficiency which at the scale of these machines can mean millions of wasted mwh of power over time.
        foobarian 3y ago
        Thanks for the detail, this is very neat. I wonder if transformers like this are in an "uncanny valley" of manufacturing where there is barely not enough demand to bootstrap mass production like for example turbines. Turbines have very exact tolerances as well, but it seems that with higher unit counts orgs like GE managed to churn them out. Perhaps the liquid piece is also making things more difficult; I'm sure a dry object is a lot easier to warehouse until needed.
        3y ago
- breezedream 3y ago
  Any thoughts on how the situation compares in Western Europe? Too many disparate systems, or…?
  - idiotsecant 3y ago
    No idea, my experience is strictly NA. I wouldn't want to speculate
- ryanblakeley 3y ago
  I appreciate your perspective. Perhaps I should have said "few" instead of "many" and "parts" instead of "systems". But it does sound like your overall impression of backup and restorative capacity differs from what I gathered in that interview. He did say damaged HV transformers would be a big deal and repairs would take weeks, but maybe that was an optimistic take. Talking about solar storms and grid damage, the range of possibilities is very large, depending on the storm's intensity and where it hits.
31carmichael 3y ago
I work in power grids, and I can also confirm that number 3 is not true. If a large number of high-capacity power transformers are required, we are in trouble.

idiotsecant 3y ago

I won't comment on 1 and 2 but 3) is laughably untrue. I guess the word 'many' might be stretched to mean 'a few' but I've worked at more than one very large utility directly as an engineer responsible for design of large generation and transmission interface systems and these transformers are not only not spare, they're almost impossible to get quickly even at current market demand. If you toasted even 10 percent of the large generation and transmission transformers in North America it would be years before the grid would even come close to working again, and that's if there was nation state level resources applied. These are specialized machines that require specialized labor and specialized materials from a specialized international supply chain to construct. I'd make the bet that total recovery would be decades.

foobarian 3y ago
This keeps coming up and I keep wondering how in the world are large transformers that complicated to build. And I don't mean that in a dismissive way but a "boy do I have a gap in my knowledge of the world" way :-)
- s1artibartfast 3y ago
  I think that is a pretty common reaction when it comes to physical/mechanical objects. Another way of thinking about it which might be helpful is. Hundreds or thousands of very talented engineers have been working non-stop for 150+ years in a transformer arms race. This has resulted in incredibly high performance designs and incredibly high requirements.
  - foobarian 3y ago
    No doubt. But same could be said of, say, automobiles, or microchips, and many other things. Even larger objects like power plant turbines seem relatively frequently manufactured. Large transformers on the other hand appear to be closer to nuclear power plants in rarity/difficulty of replacement, and as a result for me they have a very high difficulty-of-replacement-to-perceived-complexity ratio.
    - idiotsecant 3y ago
      Large transformers are built to a very specific custom specification. Each one is a custom engineered unit. A transformer is basically a sealed vessel full of volatile oil that you pass an extremely large voltage and current through. The machine has to be perfect. The smallest clearance issue, insulation problem, etc and you have a bomb. And that's just making it safe. We haven't even talked about making it effective. Transformers are the sort of think that are not rocket science to build, but every single detail is very very important. It requires a team of people to design, construct, and validate and all these people need to have the skills from doing it many times before. These people and facilities simply don't exist in large enough numbers because they are very expensive. There are also supply chain issues to consider - the steel used in transformer magnetic circuits for example is a very specific metallurgy and construction. During COVID it was quite difficult to find. Poor steel means poor efficiency which at the scale of these machines can mean millions of wasted mwh of power over time.
      - foobarian 3y ago
        Thanks for the detail, this is very neat. I wonder if transformers like this are in an "uncanny valley" of manufacturing where there is barely not enough demand to bootstrap mass production like for example turbines. Turbines have very exact tolerances as well, but it seems that with higher unit counts orgs like GE managed to churn them out. Perhaps the liquid piece is also making things more difficult; I'm sure a dry object is a lot easier to warehouse until needed.
        3y ago
breezedream 3y ago
Any thoughts on how the situation compares in Western Europe? Too many disparate systems, or…?
- idiotsecant 3y ago
  No idea, my experience is strictly NA. I wouldn't want to speculate
ryanblakeley 3y ago
I appreciate your perspective. Perhaps I should have said "few" instead of "many" and "parts" instead of "systems". But it does sound like your overall impression of backup and restorative capacity differs from what I gathered in that interview. He did say damaged HV transformers would be a big deal and repairs would take weeks, but maybe that was an optimistic take. Talking about solar storms and grid damage, the range of possibilities is very large, depending on the storm's intensity and where it hits.

31carmichael 3y ago

I work in power grids, and I can also confirm that number 3 is not true. If a large number of high-capacity power transformers are required, we are in trouble.