4 ms·

Doesn’t it feel that, since COVID, we keep hearing more and more about these colossal, systemic failures more frequently? It makes me concerned because it seems

by bmitc 4y ago

Doesn’t it feel that, since COVID, we keep hearing more and more about these colossal, systemic failures more frequently? It makes me concerned because it seems most assume that at least someone knows what’s going on. Whereas the reality is that almost no one, if anyone, actually knows what’s going on, and that we have fragile software systems “running” everything. And COVID seems to have been a real shot to the brow for many of these systems. I just worry about what hell we’re creating. Software basically captures miscommunication and poor understanding and executes it.

Karrot_Kream 4y ago
Airlines and airports have been suffering from some bad labor shortages because of the pandemic. A lot of them were let gone or furloughed during the pandemic and staffing hasn't recovered yet. That's why it's happening so often right now.
- ghaff 4y ago
  I was on a flight recently where apparently one of the flight attendants just didn't show up. It seems that they were eventually able to grab someone from the airline who was flight attendant-certified or whatever (he was just in street clothes) so we were able to go. At least the gate agent kept us informed.
kube-system 4y ago
There's a lot of people who know exactly what's going on. We built fragile systems that optimize for efficiency, except edge cases. Because edge cases are expensive to handle and usually don't happen. Except when edge cases happen, and then we have problems.
- rsj_hn 4y ago
  This is endemic to the whole world now, which has supply chains stretched across the globe in a just in time inventory system. Some economist should figure out what percentage of our productivity gains weren't really productivity gains at all, but rather risk acceptance. E.g. if we are currently producing 1000 units per year, then a new plan in which we produce 1010 units 95% of the time, 1000 units 4.9% of the time, and then 1% of the time we'll produce only 400. And just as people do price comparisons of unreliable electricity generation to reliable generation without taking the reliability into account, the various executives that signed off on this assumed (correctly) that the public would see a gain and give them the appropriate bonuses. Wall Street did a similar thing during the Great Financial Crisis. This type of analysis has already been done by Dean Baker for productivity gains due to trade deficits[1] - e.g. instead of building an iPhone, we just design the iPhone, and then call up China and ask "make us 100 million of these", and because the designers can capture so much value of the finished phone, it appears like an explosion of labor productivity. But it's all contingent on having good terms of trade and strong IP enforcement. If these were to decline, we'd capture much less value and suddenly the productivity of those designers would plummet, even though they are still doing the same work per unit of time, because productivity is measured in terms of value creation. So it could well be that on an apples to apples comparison, total productivity growth in the US from 2000 to 2020 has been flat or negative, and we've merely hidden the losses elsewhere, for example in unexpected outages, labor force problems, or periods of rapidly rising prices (revaluations). [1] https://www.cepr.net/documents/publications/productivity_2007_06.pdf https://www.cepr.net/documents/publications/productivity_200...
- bmitc 4y ago
  > There's a lot of people who know exactly what's going on. Who or where are these people? (For any industry / system.) In my experience, people understand things at a local level but quickly lose understanding at a system level. And I think there is a subtle difference from figuring out what went wrong once something did go wrong from understanding things prior to such an occurrence. Humans are pretty good at figuring out things after the fact, but we have a pretty poor track record of systematic understanding and preventing failures.
  - yourapostasy 4y ago
    > Who or where are these people? (For any industry / system.) If you hear someone on a huge incident conference call leading the capturing of new information, threading their way from one subsystem to another (most of which are outside the person's usual silo) based upon what they see, paging in teams as necessary to get the answers they need, giving a blow-by-blow account of the troubleshooting picture they're currently painting, with reasoning behind what they think might be happening, how to test that, what possible next steps might be, instantly discarding anything that doesn't move everyone closer to resolution, crafting ad hoc "good enough" tooling on the spot as needed to get enough information to make a decision but not enough to know details to the last degree, freely admitting if they're on a subsystem they know nothing about yet able to quickly grasp the essentials in minutes from domain experts to work with those domain experts to pull just the information they need...you've found one of those people. Extremely rare to find in one person, usually you see 2-3 people in such incidents splitting such a dynamic. They don't know "exactly what's going on". But they sufficiently grasp how systems work to approximate that outcome close enough for most business purposes.
  - eep_social 4y ago
    These people are SREs (source: am one) and there is a reason everyone is trying to hire a team of them. It’s rare to find much teaching on complex systems in school so the hiring pipeline is fraught with peril and low tier companies understand the buzz but not the job. In my experience SRE is mostly people wander in and realize they’re in the right place, rather than setting out to be good at the required skills like managing complex systems and large scale incident response.
troyvit 4y ago
Yeah, I feel like we're living in a shadow disaster, where everything is screwed up but nobody's talking about it. Health care centers are still completely overwhelmed with 3-8 hour waits (in small cities, not even huge cities). Call centers can't take calls. Getting into supermarkets in large cities means walking around literal piles of human feces. It just goes on and on.
- philwelch 4y ago
  > Getting into supermarkets in large cities means walking around literal piles of human feces This is almost certainly a localized phenomenon to some specific West Coast cities.
  - troyvit 4y ago
    And Denver. Definitely Denver.
bdavis__ 4y ago
optimization. every human activity has been optimized to make more money. there is no capacity to deal with exceptions.

by bmitc 4y ago

Karrot_Kream 4y ago

Airlines and airports have been suffering from some bad labor shortages because of the pandemic. A lot of them were let gone or furloughed during the pandemic and staffing hasn't recovered yet. That's why it's happening so often right now.

ghaff 4y ago
I was on a flight recently where apparently one of the flight attendants just didn't show up. It seems that they were eventually able to grab someone from the airline who was flight attendant-certified or whatever (he was just in street clothes) so we were able to go. At least the gate agent kept us informed.

kube-system 4y ago

There's a lot of people who know exactly what's going on. We built fragile systems that optimize for efficiency, except edge cases. Because edge cases are expensive to handle and usually don't happen. Except when edge cases happen, and then we have problems.

rsj_hn 4y ago
This is endemic to the whole world now, which has supply chains stretched across the globe in a just in time inventory system. Some economist should figure out what percentage of our productivity gains weren't really productivity gains at all, but rather risk acceptance. E.g. if we are currently producing 1000 units per year, then a new plan in which we produce 1010 units 95% of the time, 1000 units 4.9% of the time, and then 1% of the time we'll produce only 400. And just as people do price comparisons of unreliable electricity generation to reliable generation without taking the reliability into account, the various executives that signed off on this assumed (correctly) that the public would see a gain and give them the appropriate bonuses. Wall Street did a similar thing during the Great Financial Crisis. This type of analysis has already been done by Dean Baker for productivity gains due to trade deficits[1] - e.g. instead of building an iPhone, we just design the iPhone, and then call up China and ask "make us 100 million of these", and because the designers can capture so much value of the finished phone, it appears like an explosion of labor productivity. But it's all contingent on having good terms of trade and strong IP enforcement. If these were to decline, we'd capture much less value and suddenly the productivity of those designers would plummet, even though they are still doing the same work per unit of time, because productivity is measured in terms of value creation. So it could well be that on an apples to apples comparison, total productivity growth in the US from 2000 to 2020 has been flat or negative, and we've merely hidden the losses elsewhere, for example in unexpected outages, labor force problems, or periods of rapidly rising prices (revaluations). [1] https://www.cepr.net/documents/publications/productivity_2007_06.pdf https://www.cepr.net/documents/publications/productivity_200...
bmitc 4y ago
> There's a lot of people who know exactly what's going on. Who or where are these people? (For any industry / system.) In my experience, people understand things at a local level but quickly lose understanding at a system level. And I think there is a subtle difference from figuring out what went wrong once something did go wrong from understanding things prior to such an occurrence. Humans are pretty good at figuring out things after the fact, but we have a pretty poor track record of systematic understanding and preventing failures.
- yourapostasy 4y ago
  > Who or where are these people? (For any industry / system.) If you hear someone on a huge incident conference call leading the capturing of new information, threading their way from one subsystem to another (most of which are outside the person's usual silo) based upon what they see, paging in teams as necessary to get the answers they need, giving a blow-by-blow account of the troubleshooting picture they're currently painting, with reasoning behind what they think might be happening, how to test that, what possible next steps might be, instantly discarding anything that doesn't move everyone closer to resolution, crafting ad hoc "good enough" tooling on the spot as needed to get enough information to make a decision but not enough to know details to the last degree, freely admitting if they're on a subsystem they know nothing about yet able to quickly grasp the essentials in minutes from domain experts to work with those domain experts to pull just the information they need...you've found one of those people. Extremely rare to find in one person, usually you see 2-3 people in such incidents splitting such a dynamic. They don't know "exactly what's going on". But they sufficiently grasp how systems work to approximate that outcome close enough for most business purposes.
- eep_social 4y ago
  These people are SREs (source: am one) and there is a reason everyone is trying to hire a team of them. It’s rare to find much teaching on complex systems in school so the hiring pipeline is fraught with peril and low tier companies understand the buzz but not the job. In my experience SRE is mostly people wander in and realize they’re in the right place, rather than setting out to be good at the required skills like managing complex systems and large scale incident response.

troyvit 4y ago

Yeah, I feel like we're living in a shadow disaster, where everything is screwed up but nobody's talking about it. Health care centers are still completely overwhelmed with 3-8 hour waits (in small cities, not even huge cities). Call centers can't take calls. Getting into supermarkets in large cities means walking around literal piles of human feces. It just goes on and on.

philwelch 4y ago
> Getting into supermarkets in large cities means walking around literal piles of human feces This is almost certainly a localized phenomenon to some specific West Coast cities.
- troyvit 4y ago
  And Denver. Definitely Denver.

bdavis__ 4y ago

optimization. every human activity has been optimized to make more money. there is no capacity to deal with exceptions.