4 ms·

You wouldn’t believe how fragile some of our supply chains are, and how many single points of failure we have. For a while a decade and a bit ago, there was on

You wouldn’t believe how fragile some of our supply chains are, and how many single points of failure we have. For a while a decade and a bit ago, there was only one operational, legal tantalum smelter on the planet - the others had been destroyed by the Indian Ocean tsunami or shut down over their use of conflict minerals and slavery. I think there are now three - all on the ring of fire. Then you’ve got stuff like cobalt, which we get at the whim of the DRC, a notoriously stable and functional state. And of course, gallium, graphite, and rare earths, which pretty much all come from China. They could, at will, bring the global electronics supply chain to its knees. I guess risk is everywhere, really, in our fragile systems.

AdamN 2y ago
One advantage of these types of things is how easy they are to stockpile. Aside from the nominal storage costs, none of these things degrade over time: https://www.dla.mil/Strategic-Materials/ https://www.dla.mil/Strategic-Materials/
- mschuster91 2y ago
  Stockpiles and other forms of resiliency (such as redundant facilities) cost a lot of money because by definition you cannot run them at nominal capacity or it's just a ton of expensive stuff that sits around. Capitalism is fundamentally incapable of accounting for resiliency as it optimizes to cut such "dead costs".
  - lancebeet 2y ago
    Toyota, who is widely recognized as the pioneer of JIT manufacturing, famously began stockpiling microchips after the Fukushima disaster disrupted their supply chains. It might not make sense to stockpile raw materials, but stockpiling things with comparatively low storage costs w.r.t. the item cost sometimes makes sense from a capitalist perspective. This assumes the company runs with a long-term perspective in mind, which often isn't the case in western companies that are publicly traded or owned/operated by investment firms. In fairness, governments also frequently fail at accounting for resiliency, even when that's their explicit goal, so perhaps the incapability to account for resiliency goes beyond the economic system.
    - mschuster91 2y ago
      > In fairness, governments also frequently fail at accounting for resiliency, even when that's their explicit goal, so perhaps the incapability to account for resiliency goes beyond the economic system. That's because governments are run by the same braindead efficiency-first mindset these days. It used to be easy to campaign on resiliency back when the USSR still existed - no matter what, all NATO countries had really large armies, months worth of stockpiles of everything from arms and munitions over medical supplies to food. Education was heavily invested in because the countries needed well educated adults to compete with the external threat. Companies kept stockpiles and had detailed disaster recovery plans and drills, not just for natural disasters but also for "how the fuck do we keep production running when the Red Army is marching towards the border" and "how the fuck do we transition from building cars and motorcycles to tanks and fighter planes when the Great War breaks out". But once the USSR fell and the "end of history" was announced [1], all of that resiliency went down the drain - it was expensive after all and neoliberalism / "starve the beast" ideologues demanded that the "inefficiency" be cut to lower the tax burden on the corporations and the rich... and here we are, our education systems in shambles, companies got screwed left and right when covid hit and supply chains broke, and our societies in shambles as well as the people didn't have much reserves left after decades of wage stagnation and companies didn't either so they laid people off. [1] https://en.wikipedia.org/wiki/The_End_of_History_and_the_Last_Man https://en.wikipedia.org/wiki/The_End_of_History_and_the_Las...
    - p_l 2y ago
      What a lot of people (and companies) missed is that Toyota doesn't optimize till there's no slack left, and instead properly run Kanban let's you establish a capacity and slack "reserve" in the system. Some people seem to run companies towards absolutely no slack at all, and are baffled by the idea of keeping some.
    - ahartmetz 2y ago
      > the incapability to account for resiliency goes beyond the economic system Reminds me of how bad pollution was in the GDR (communist Germany). You think capitalism is bad for the environment? Communism was worse.
  - zmgsabst 2y ago
    I don’t think that’s a fundamental inability; we just choose not to due to fashion in business thought. But there’s nothing fundamentally that prevents pricing in risk of shortages to shareholders and appropriately stockpiling to mitigate that risk.
    - mschuster91 2y ago
      > But there’s nothing fundamentally that prevents pricing in risk of shortages to shareholders and appropriately stockpiling to mitigate that risk. There is. Shareholders are just as likely to fall into the short-term gains trap.
  - gen220 2y ago
    IDK, Berkshire Hathaway sure has a big stockpile of cash, and they're emblematic of capitalism. The cash in this model would be considered a "dead cost", as it's "wasted" by not being immediately invested in equities. I agree with you, in general, that the short-term incentives of our system encourage "optimization" away from stockpiling inputs, and that this is a worrying problem. But I find it instructive to focus on and give credit to the long-term thinkers who buck those trends – and who have historically profited and persisted by doing so, by the way.
crote 2y ago
I vaguely remember a story about an accidental single point of failure. If I recall correctly, during the 1970s or 1980 there was a critical part of the wafer production process (slicing the crystal into wafers, perhaps?) which relied on highly specialized tooling. As it turns out, a bunch of manufacturers had independently discovered that by far the best source of them was a small company in Japan - unbeknownst to them essentially just one guy and his workshop. So everyone sourced it from him. Which was fine, until he took a holiday for the first time in a decade... Suddenly both original-source and second-source factories for some important chips were unavailable! I'd love to know if anyone has more details, because I haven't had any luck with finding any reference to it. It's a story I read probably a decade ago, so who knows how many details I got wrong.
- sanxiyn 2y ago
  Sounds like DISCO to me, although details differ.
pjc50 2y ago
Everyone seems to forget that Silicon Valley and its datacenters are in an earthquake zone.
- samus 2y ago
  A typical multinational social media operator has datacenters around the globe. The question is rather whether these can really function independently should the central ones go offline.
- ElevenLathe 2y ago
  I would be more worried about Northern Virginia. I doubt the world at large would notice much problem if SFBA dropped off the network, but they sure would if the same happened to NoVA.
- p_l 2y ago
  AFAIK till quite recently, if not still, us-east-1 was SPOF for various components of AWS. Your already deployed things would continue to work, but APIs (and most importantly, Console) wouldn't, IIRC.
deleted 2y ago
[deleted]
chasil 2y ago
Half of the world's neon was coming from Ukraine, a quarter each from Odessa and Mariupol. This is also critical for semiconductor manufacturing. https://www.theregister.com/2022/03/11/ukraine_neon_supplies/ https://www.theregister.com/2022/03/11/ukraine_neon_supplies...

AdamN 2y ago

One advantage of these types of things is how easy they are to stockpile. Aside from the nominal storage costs, none of these things degrade over time: https://www.dla.mil/Strategic-Materials/ https://www.dla.mil/Strategic-Materials/

mschuster91 2y ago
Stockpiles and other forms of resiliency (such as redundant facilities) cost a lot of money because by definition you cannot run them at nominal capacity or it's just a ton of expensive stuff that sits around. Capitalism is fundamentally incapable of accounting for resiliency as it optimizes to cut such "dead costs".
- lancebeet 2y ago
  Toyota, who is widely recognized as the pioneer of JIT manufacturing, famously began stockpiling microchips after the Fukushima disaster disrupted their supply chains. It might not make sense to stockpile raw materials, but stockpiling things with comparatively low storage costs w.r.t. the item cost sometimes makes sense from a capitalist perspective. This assumes the company runs with a long-term perspective in mind, which often isn't the case in western companies that are publicly traded or owned/operated by investment firms. In fairness, governments also frequently fail at accounting for resiliency, even when that's their explicit goal, so perhaps the incapability to account for resiliency goes beyond the economic system.
  - mschuster91 2y ago
    > In fairness, governments also frequently fail at accounting for resiliency, even when that's their explicit goal, so perhaps the incapability to account for resiliency goes beyond the economic system. That's because governments are run by the same braindead efficiency-first mindset these days. It used to be easy to campaign on resiliency back when the USSR still existed - no matter what, all NATO countries had really large armies, months worth of stockpiles of everything from arms and munitions over medical supplies to food. Education was heavily invested in because the countries needed well educated adults to compete with the external threat. Companies kept stockpiles and had detailed disaster recovery plans and drills, not just for natural disasters but also for "how the fuck do we keep production running when the Red Army is marching towards the border" and "how the fuck do we transition from building cars and motorcycles to tanks and fighter planes when the Great War breaks out". But once the USSR fell and the "end of history" was announced [1], all of that resiliency went down the drain - it was expensive after all and neoliberalism / "starve the beast" ideologues demanded that the "inefficiency" be cut to lower the tax burden on the corporations and the rich... and here we are, our education systems in shambles, companies got screwed left and right when covid hit and supply chains broke, and our societies in shambles as well as the people didn't have much reserves left after decades of wage stagnation and companies didn't either so they laid people off. [1] https://en.wikipedia.org/wiki/The_End_of_History_and_the_Last_Man https://en.wikipedia.org/wiki/The_End_of_History_and_the_Las...
  - p_l 2y ago
    What a lot of people (and companies) missed is that Toyota doesn't optimize till there's no slack left, and instead properly run Kanban let's you establish a capacity and slack "reserve" in the system. Some people seem to run companies towards absolutely no slack at all, and are baffled by the idea of keeping some.
  - ahartmetz 2y ago
    > the incapability to account for resiliency goes beyond the economic system Reminds me of how bad pollution was in the GDR (communist Germany). You think capitalism is bad for the environment? Communism was worse.
- zmgsabst 2y ago
  I don’t think that’s a fundamental inability; we just choose not to due to fashion in business thought. But there’s nothing fundamentally that prevents pricing in risk of shortages to shareholders and appropriately stockpiling to mitigate that risk.
  - mschuster91 2y ago
    > But there’s nothing fundamentally that prevents pricing in risk of shortages to shareholders and appropriately stockpiling to mitigate that risk. There is. Shareholders are just as likely to fall into the short-term gains trap.
- gen220 2y ago
  IDK, Berkshire Hathaway sure has a big stockpile of cash, and they're emblematic of capitalism. The cash in this model would be considered a "dead cost", as it's "wasted" by not being immediately invested in equities. I agree with you, in general, that the short-term incentives of our system encourage "optimization" away from stockpiling inputs, and that this is a worrying problem. But I find it instructive to focus on and give credit to the long-term thinkers who buck those trends – and who have historically profited and persisted by doing so, by the way.

crote 2y ago

I vaguely remember a story about an accidental single point of failure. If I recall correctly, during the 1970s or 1980 there was a critical part of the wafer production process (slicing the crystal into wafers, perhaps?) which relied on highly specialized tooling. As it turns out, a bunch of manufacturers had independently discovered that by far the best source of them was a small company in Japan - unbeknownst to them essentially just one guy and his workshop. So everyone sourced it from him. Which was fine, until he took a holiday for the first time in a decade... Suddenly both original-source and second-source factories for some important chips were unavailable! I'd love to know if anyone has more details, because I haven't had any luck with finding any reference to it. It's a story I read probably a decade ago, so who knows how many details I got wrong.

sanxiyn 2y ago
Sounds like DISCO to me, although details differ.

pjc50 2y ago

Everyone seems to forget that Silicon Valley and its datacenters are in an earthquake zone.

samus 2y ago
A typical multinational social media operator has datacenters around the globe. The question is rather whether these can really function independently should the central ones go offline.
ElevenLathe 2y ago
I would be more worried about Northern Virginia. I doubt the world at large would notice much problem if SFBA dropped off the network, but they sure would if the same happened to NoVA.
p_l 2y ago
AFAIK till quite recently, if not still, us-east-1 was SPOF for various components of AWS. Your already deployed things would continue to work, but APIs (and most importantly, Console) wouldn't, IIRC.

deleted 2y ago

[deleted]

chasil 2y ago

Half of the world's neon was coming from Ukraine, a quarter each from Odessa and Mariupol. This is also critical for semiconductor manufacturing. https://www.theregister.com/2022/03/11/ukraine_neon_supplies/ https://www.theregister.com/2022/03/11/ukraine_neon_supplies...