4 ms·

It's absolutely ludicrous the US airline industry still has no contingency plan for "a bad software update is pushed by a vendor to our Windows systems."

SketchySeaBeast 2y ago
If the computers in a computer reliant massively distributed organization go down, what's the alternative?
- ThunderSizzle 2y ago
  Computers shouldn't auto update. Updates need to be scheduled and monitored and tested.
  - lazide 2y ago
    Every CTO’s easy answer - cool, we’ll just only do them once every 5 years then. With modern tech stacks, we’re talking hundreds of updates a quarter. If not more.
    - cozzyd 2y ago
      Sounds like they shouldn't be using those tech sticks then.
  - paulmd 2y ago
    That was the defacto state of things up until windows 10 moved to a forced update model. In that world we regularly had massive malware waves and resultant massive botnets constantly disrupting critical infrastructure. We moved away from that for a reason. And the real fix is to make updates not take down the whole system. When was the last time every Linux system in the world was unable to boot all at the same time? Even the crowdstrike issue only took down a very limited subset of installs and the blast radius is far more minimal due to the linux kernel being a lot more sane and cautious.
  - paulddraper 2y ago
    Desktop operating systems, mobile devices, browsers.....there's a lot that doesn't pass your approval.
    - ThunderSizzle 2y ago
      My desktop doesn't auto update. It only updates when I ask it to. My phone does auto update, and nearly every auto update makes it worse and slower. That's been true for nearly every Android I've had.
      - paulddraper 2y ago
        The sheer awesomeness of Android 3 is staggering by now.
- unstatusthequo 2y ago
  Failover?
  - SketchySeaBeast 2y ago
    Failover to what?
- amluto 2y ago
  Acknowledge that it’s a distributed system and build accordingly. The distributed systems literature understands failure modes, including Byzantine failure. Old airline systems were built to work with simple protocols and limited communication. A ticket was an independently verifiable object. Reservations were on a printed list. Check-in and gate agents did not need to make a phone call for each passenger. This was expensive, but it worked. One can do the same thing with computers. Build small systems that operate independently, with strictly defined inputs and outputs. Make them un-brickable. Distribute software on read-only media, or outright replace the CPU for each update. You roll back by having the staff unplug the new thing and plug in the old thing. A high-spec CM4 is $90. A small USB stick costs basically nothing. A bespoke CM4-like machine with only ROM could be made in volume for relatively little money. Most of the problem with an approach like this is that the industry doesn’t think this way. Maybe a big buyer should step up, e.g. a major military. Non-CrowdStrike-able tech for, say, the US military seems like it would be extremely valuable.
- jjav 2y ago
  > If the computers in a computer reliant massively distributed organization go down, what's the alternative? I feel like the answer should be obvious: if you need reliable systems then you don't create a single point of dependency on such fragile systems. The latest blog from Bruce Schneier is a good writeup on this brittleness: https://www.schneier.com/blog/archives/2024/07/the-crowdstrike-outage-and-market-driven-brittleness.html https://www.schneier.com/blog/archives/2024/07/the-crowdstri...

SketchySeaBeast 2y ago

If the computers in a computer reliant massively distributed organization go down, what's the alternative?

ThunderSizzle 2y ago
Computers shouldn't auto update. Updates need to be scheduled and monitored and tested.
- lazide 2y ago
  Every CTO’s easy answer - cool, we’ll just only do them once every 5 years then. With modern tech stacks, we’re talking hundreds of updates a quarter. If not more.
  - cozzyd 2y ago
    Sounds like they shouldn't be using those tech sticks then.
- paulmd 2y ago
  That was the defacto state of things up until windows 10 moved to a forced update model. In that world we regularly had massive malware waves and resultant massive botnets constantly disrupting critical infrastructure. We moved away from that for a reason. And the real fix is to make updates not take down the whole system. When was the last time every Linux system in the world was unable to boot all at the same time? Even the crowdstrike issue only took down a very limited subset of installs and the blast radius is far more minimal due to the linux kernel being a lot more sane and cautious.
- paulddraper 2y ago
  Desktop operating systems, mobile devices, browsers.....there's a lot that doesn't pass your approval.
  - ThunderSizzle 2y ago
    My desktop doesn't auto update. It only updates when I ask it to. My phone does auto update, and nearly every auto update makes it worse and slower. That's been true for nearly every Android I've had.
    - paulddraper 2y ago
      The sheer awesomeness of Android 3 is staggering by now.
unstatusthequo 2y ago
Failover?
- SketchySeaBeast 2y ago
  Failover to what?
amluto 2y ago
Acknowledge that it’s a distributed system and build accordingly. The distributed systems literature understands failure modes, including Byzantine failure. Old airline systems were built to work with simple protocols and limited communication. A ticket was an independently verifiable object. Reservations were on a printed list. Check-in and gate agents did not need to make a phone call for each passenger. This was expensive, but it worked. One can do the same thing with computers. Build small systems that operate independently, with strictly defined inputs and outputs. Make them un-brickable. Distribute software on read-only media, or outright replace the CPU for each update. You roll back by having the staff unplug the new thing and plug in the old thing. A high-spec CM4 is $90. A small USB stick costs basically nothing. A bespoke CM4-like machine with only ROM could be made in volume for relatively little money. Most of the problem with an approach like this is that the industry doesn’t think this way. Maybe a big buyer should step up, e.g. a major military. Non-CrowdStrike-able tech for, say, the US military seems like it would be extremely valuable.
jjav 2y ago
> If the computers in a computer reliant massively distributed organization go down, what's the alternative? I feel like the answer should be obvious: if you need reliable systems then you don't create a single point of dependency on such fragile systems. The latest blog from Bruce Schneier is a good writeup on this brittleness: https://www.schneier.com/blog/archives/2024/07/the-crowdstrike-outage-and-market-driven-brittleness.html https://www.schneier.com/blog/archives/2024/07/the-crowdstri...