5 ms·

> is the only one worth going into Sure, when your critical system goes down an RCA is hugely important and ultimately you have to apply a fix that addresses t

> is the only one worth going into Sure, when your critical system goes down an RCA is hugely important and ultimately you have to apply a fix that addresses the core issue to avoid it happening again in the future. But, at the time that the system is actually down it seems like the most important first step (once you understand the problem) is to get the system running again ASAP. This can give you the runway to fix the actual problem.

gkop 5y ago
Counterpoint, sometimes it's better to let the system burn, or else the root cause will never be addressed. Treating the symptoms can take the pressure off solving the root cause.
- hbosch 5y ago
  Correct! If we simply let the person die of cancer, we can properly investigate the tumor when they are dead.
  - gkop 5y ago
    I should have given more context. In cases where incentives are deeply, structurally misaligned, and it will take heroic effort and significant luck to yield an order of magnitude improvement over the status quo, we should consider "letting it burn" as an option, and recognize the total cost of treating the symptoms. The global logistics quagmire may be a candidate for nuclear-ish options. Agree with you on the cancer patient scenario.
    - labster 5y ago
      Let the global supply chain burn? By Jove, let’s have a great depression!
      - gkop 5y ago
        Let the containers on the streets piss people off to build pressure to align incentives, rather than prolonging the problem with a temporary stacking improvement. This is just a tool in our toolbox that we should not ignore, I'm not saying it's the right tool. But there is a cost of papering over the root cause, that's not free. BTW I don't live in LA/Long Beach. I recognize that LA doesn't deserve the quality of life degradation, that's an externality. We have tools to resolve externalities. I could imagine living in an affected neighborhood in LA and being super grateful for the container stacking "quick fix".
        labster 5y ago
        But the temporary rule change is a quality of life improvement. Less trucks idling on the streets, less ships idling offshore means better air quality. Less ships anchored means less chance of another oil pipeline spill. More cargo hauled means more prosperity. A few months of eyesore are an acceptable cost. The thing is, there is no fix to the root cause. You can either have cheaper products with just-in-time supply chains, or you can pay more for storage. The trade-off will always be efficiency or redundancy, and most industries have already chosen the level of risk they can accept. Real world systems have tipping points and bottlenecks, and it’s okay to use government to push them back into steady state.
      - bregma 5y ago
        Why not? It worked last time.
    - kelnos 5y ago
      The problem is that often the consequences of letting it burn are most felt by innocent bystanders, rather than the people who are meant to be "taught a lesson".
neltnerb 5y ago
It sounded like the capacity to physically move things around is being blocked because trucks are being used for storage. In that particular situation a temporary buffer that allows the flow to become unblocked is necessary. The computer won't operate if you are unable to move data off the internal registers because there's nowhere to go. Including operations to delete the data in long term storage that is preventing the internal registers from being cleared.
SkeuomorphicBee 5y ago
For us software developers, it seems like the shipping industry has the container equivalent of a memory leak (a "container leak" if you will). Then the stacking rule change is the equivalent of simply adding more memory to the system, it doesn't fix the problem, but it buys you some more time of normal system operation before the next out-of-memory crash. Hopefully they use this time they bought to work on an actual solution to the original leak.
- bluGill 5y ago
  No, it isn't a leak as the containers are empty but not garbage. It is very common to allocate extra memory that isn't used - in garbage collected languages you often need to fight the garbage collector for maximum performance in specific ways - which means you will have empty objects just waiting to be filled. You will get around to them eventually, but for now they are just taking up memory. If you allocate more of these objects than you have physical ram you will start swapping - but there is no leak, you will eventually either use them, or destroy them. If you don't worth in latency senstive applications you might not have encountered a situation where you need to apply the above tricks.
  - tuatoru 5y ago
    Lol! Continuing this analogy, what has happened is that the swap space for containers has filled, and it now has a form of compression applied to it, so that five containers can now be stored in the space where two could be beforehand. (Edit: let's hope the swap space doesn't become encrypted.) I wonder when the out-of-memory-killer process will start up? What would it look like--just not shipping anything to the US for a few months?
- morelisp 5y ago
  If you want to phrase it in programming terms, there's a pretty obvious comparison to semaphores and deadlocks. We've deadlocked. As a workaround we've increased a bunch of our semaphore limits. But we also had those limits for a reason (various balances of safety, efficiency, and available amount of other ancillary resources). Maybe now we'll run out of some other resource, or hit contention somewhere else - we don't really know, the system is big and hasn't been operated in this state before. And on top of that, we didn't solve the fundamental deadlock - if the underlying conditions persist, there's hard limits to how often we can do this before we deadlock permanently.
mcguire 5y ago
Assuming no other conditions change, how long will it take them to use up the extra storage space?

gkop 5y ago

Counterpoint, sometimes it's better to let the system burn, or else the root cause will never be addressed. Treating the symptoms can take the pressure off solving the root cause.

hbosch 5y ago
Correct! If we simply let the person die of cancer, we can properly investigate the tumor when they are dead.
- gkop 5y ago
  I should have given more context. In cases where incentives are deeply, structurally misaligned, and it will take heroic effort and significant luck to yield an order of magnitude improvement over the status quo, we should consider "letting it burn" as an option, and recognize the total cost of treating the symptoms. The global logistics quagmire may be a candidate for nuclear-ish options. Agree with you on the cancer patient scenario.
  - labster 5y ago
    Let the global supply chain burn? By Jove, let’s have a great depression!
    - gkop 5y ago
      Let the containers on the streets piss people off to build pressure to align incentives, rather than prolonging the problem with a temporary stacking improvement. This is just a tool in our toolbox that we should not ignore, I'm not saying it's the right tool. But there is a cost of papering over the root cause, that's not free. BTW I don't live in LA/Long Beach. I recognize that LA doesn't deserve the quality of life degradation, that's an externality. We have tools to resolve externalities. I could imagine living in an affected neighborhood in LA and being super grateful for the container stacking "quick fix".
      - labster 5y ago
        But the temporary rule change is a quality of life improvement. Less trucks idling on the streets, less ships idling offshore means better air quality. Less ships anchored means less chance of another oil pipeline spill. More cargo hauled means more prosperity. A few months of eyesore are an acceptable cost. The thing is, there is no fix to the root cause. You can either have cheaper products with just-in-time supply chains, or you can pay more for storage. The trade-off will always be efficiency or redundancy, and most industries have already chosen the level of risk they can accept. Real world systems have tipping points and bottlenecks, and it’s okay to use government to push them back into steady state.
    - bregma 5y ago
      Why not? It worked last time.
  - kelnos 5y ago
    The problem is that often the consequences of letting it burn are most felt by innocent bystanders, rather than the people who are meant to be "taught a lesson".

neltnerb 5y ago

It sounded like the capacity to physically move things around is being blocked because trucks are being used for storage. In that particular situation a temporary buffer that allows the flow to become unblocked is necessary. The computer won't operate if you are unable to move data off the internal registers because there's nowhere to go. Including operations to delete the data in long term storage that is preventing the internal registers from being cleared.

SkeuomorphicBee 5y ago

For us software developers, it seems like the shipping industry has the container equivalent of a memory leak (a "container leak" if you will). Then the stacking rule change is the equivalent of simply adding more memory to the system, it doesn't fix the problem, but it buys you some more time of normal system operation before the next out-of-memory crash. Hopefully they use this time they bought to work on an actual solution to the original leak.

bluGill 5y ago
No, it isn't a leak as the containers are empty but not garbage. It is very common to allocate extra memory that isn't used - in garbage collected languages you often need to fight the garbage collector for maximum performance in specific ways - which means you will have empty objects just waiting to be filled. You will get around to them eventually, but for now they are just taking up memory. If you allocate more of these objects than you have physical ram you will start swapping - but there is no leak, you will eventually either use them, or destroy them. If you don't worth in latency senstive applications you might not have encountered a situation where you need to apply the above tricks.
- tuatoru 5y ago
  Lol! Continuing this analogy, what has happened is that the swap space for containers has filled, and it now has a form of compression applied to it, so that five containers can now be stored in the space where two could be beforehand. (Edit: let's hope the swap space doesn't become encrypted.) I wonder when the out-of-memory-killer process will start up? What would it look like--just not shipping anything to the US for a few months?
morelisp 5y ago
If you want to phrase it in programming terms, there's a pretty obvious comparison to semaphores and deadlocks. We've deadlocked. As a workaround we've increased a bunch of our semaphore limits. But we also had those limits for a reason (various balances of safety, efficiency, and available amount of other ancillary resources). Maybe now we'll run out of some other resource, or hit contention somewhere else - we don't really know, the system is big and hasn't been operated in this state before. And on top of that, we didn't solve the fundamental deadlock - if the underlying conditions persist, there's hard limits to how often we can do this before we deadlock permanently.

mcguire 5y ago

Assuming no other conditions change, how long will it take them to use up the extra storage space?