3 ms·

The Microsoft part of this I find interesting, I mean I guess it would be standard legal practice to also go after them even if it is thrown out. But that one i

The Microsoft part of this I find interesting, I mean I guess it would be standard legal practice to also go after them even if it is thrown out. But that one is weird. Maybe Microsoft encouraged Delta to use Crowdstrike, then ok I could see the case. But if it is anything related to how Windows works, that seems like a stretch. Yes Windows could be better, but I don't like the idea that the OS provider could be sued for a piece of software causing problems. That seems... dangerous. Regarding Delta. This seems like an interesting situation. Apparently Crowdstrike offered people to help, but how many people and where. Realistically how much help would they have been able to help given that it isn't like this was isolated to Delta and I imagine their people were stretched pretty thin. And importantly what exactly was the help provided. I don't see any information about this. While I am sure that Delta's IT department was understaffed, this was also a unique situation. If you spent the time to make a well optimized machine for rolling out updates, things were automated, and you expected things to go wrong but I would have never anticipated every Windows machine being unable to boot. That is an extra-ordinary situation. I doubt any IT department is really staffed to be able to handle that situation happening. You don't expect to need to deal with every machine you have, its honestly kinda unrealistic. It will be interesting to watch this one play out in the courts. Because at the end of the day the vast majority of the blame for this entire situation is on Crowdstrike. The companies were handed a situation that we were not prepared for.

beanjuiceII 2y ago
delta is responsible for delta, there are no absolutes in software and systems, at the end of the day delta has to be ready for anything, if they choose to under staff and rely on black box software..that is a choice they make. Microsoft was forced by EU, and crowdstrike had a (rather large) mistake, companies have them you know. It's the same reason why i still backup my google photos and drive.
3np 2y ago
> While I am sure that Delta's IT department was understaffed, this was also a unique situation. If you spent the time to make a well optimized machine for rolling out updates, things were automated, and you expected things to go wrong but I would have never anticipated every Windows machine being unable to boot. That is an extra-ordinary situation. I doubt any IT department is really staffed to be able to handle that situation happening. You don't expect to need to deal with every machine you have, its honestly kinda unrealistic. I do not believe this will be the last of its nature in our generation. It should very much be already have been accounted for. The basics and fundamentals of information security apply despite what interpretations of insurance policies or certifications say. Let this be a lesson to not introduce single points of failure into critical systems without having prepared for their unavailaility or misbehaviour (yes, that includes your ISPs, cloud, and SaaS providers
- nerdjon 2y ago
  > Let this be a lesson to not introduce single points of failure into critical systems without having prepared for their unavailaility or misbehaviour. I think that is where this is going to get more complicated, I think that this has broadened what we traditionally think about when it comes to what we consider a single point of failure. I would wager that most people would not have considered something like Crowdstrike to be a single point of failure. It is not what you would generally think of. From what sounds of the bits and pieces of Delta that came out, it sounds like the biggest issue they had was that so many of these systems were not together. It required physical access to systems that were in many different locations (I would love to be corrected on this). Under normal circumstances that would cover you in the case of an outage in a particular location. If you built your system to handle entire areas going down, adding in redundancy. But this was very much a unique situation. We really shouldn't pretend otherwise. It wasn't an external service going down, or AWS going down, loss of internet connection, or other things going down that we would normally account for in disaster recovery. Hell I would argue that in this particular situation, you could have the best disaster recovery but it wouldn't have done anything since Crowdstrike was probably baked into your images. So have fun bringing up an instance that was going to instantly brick. Yeah eventually that would have been a non issue, but I can not imagine a scenario that I would have ever thought that I had a baked image and assuming I still had access to it and I could assure that it was the image that I made, that it somehow would itself be a problem. Before a couple weeks ago if we had built a system that was redundant enough to handle parts going down, different geographical locations, etc etc with, maybe things slow down but still works. All of the standard things that we talk about. It isn't absurd to think that your entire system going down in a situation where you don't think you have a single point of failure, is nearly impossible or if it happens there is something outside of your company seriously bad.
  - 3np 2y ago
    > I would wager that most people would not have considered something like Crowdstrike to be a single point of failure. It is not what you would generally think of. Sure. Most people are not decision-makers for critical IT infrastructure. Should we similarly throw our hands in the air if a bridge or building collapses because the people responsible didn't pay attention to structural safety just because it wasn't obvious to a layperson? > Hell I would argue that in this particular situation, you could have the best disaster recovery but it wouldn't have done anything since Crowdstrike was probably baked into your images. It's not really "the best disaster recovery" then, is it? > Yeah eventually that would have been a non issue, but I can not imagine a scenario that I would have ever thought that I had a baked image and assuming I still had access to it and I could assure that it was the image that I made, that it somehow would itself be a problem. There are so many other ways this can back-fire I'm not going to try to enumerate them. I really hope no one put you in charge of operating anything critical...

beanjuiceII 2y ago

delta is responsible for delta, there are no absolutes in software and systems, at the end of the day delta has to be ready for anything, if they choose to under staff and rely on black box software..that is a choice they make. Microsoft was forced by EU, and crowdstrike had a (rather large) mistake, companies have them you know. It's the same reason why i still backup my google photos and drive.

3np 2y ago

> While I am sure that Delta's IT department was understaffed, this was also a unique situation. If you spent the time to make a well optimized machine for rolling out updates, things were automated, and you expected things to go wrong but I would have never anticipated every Windows machine being unable to boot. That is an extra-ordinary situation. I doubt any IT department is really staffed to be able to handle that situation happening. You don't expect to need to deal with every machine you have, its honestly kinda unrealistic. I do not believe this will be the last of its nature in our generation. It should very much be already have been accounted for. The basics and fundamentals of information security apply despite what interpretations of insurance policies or certifications say. Let this be a lesson to not introduce single points of failure into critical systems without having prepared for their unavailaility or misbehaviour (yes, that includes your ISPs, cloud, and SaaS providers

nerdjon 2y ago
> Let this be a lesson to not introduce single points of failure into critical systems without having prepared for their unavailaility or misbehaviour. I think that is where this is going to get more complicated, I think that this has broadened what we traditionally think about when it comes to what we consider a single point of failure. I would wager that most people would not have considered something like Crowdstrike to be a single point of failure. It is not what you would generally think of. From what sounds of the bits and pieces of Delta that came out, it sounds like the biggest issue they had was that so many of these systems were not together. It required physical access to systems that were in many different locations (I would love to be corrected on this). Under normal circumstances that would cover you in the case of an outage in a particular location. If you built your system to handle entire areas going down, adding in redundancy. But this was very much a unique situation. We really shouldn't pretend otherwise. It wasn't an external service going down, or AWS going down, loss of internet connection, or other things going down that we would normally account for in disaster recovery. Hell I would argue that in this particular situation, you could have the best disaster recovery but it wouldn't have done anything since Crowdstrike was probably baked into your images. So have fun bringing up an instance that was going to instantly brick. Yeah eventually that would have been a non issue, but I can not imagine a scenario that I would have ever thought that I had a baked image and assuming I still had access to it and I could assure that it was the image that I made, that it somehow would itself be a problem. Before a couple weeks ago if we had built a system that was redundant enough to handle parts going down, different geographical locations, etc etc with, maybe things slow down but still works. All of the standard things that we talk about. It isn't absurd to think that your entire system going down in a situation where you don't think you have a single point of failure, is nearly impossible or if it happens there is something outside of your company seriously bad.
- 3np 2y ago
  > I would wager that most people would not have considered something like Crowdstrike to be a single point of failure. It is not what you would generally think of. Sure. Most people are not decision-makers for critical IT infrastructure. Should we similarly throw our hands in the air if a bridge or building collapses because the people responsible didn't pay attention to structural safety just because it wasn't obvious to a layperson? > Hell I would argue that in this particular situation, you could have the best disaster recovery but it wouldn't have done anything since Crowdstrike was probably baked into your images. It's not really "the best disaster recovery" then, is it? > Yeah eventually that would have been a non issue, but I can not imagine a scenario that I would have ever thought that I had a baked image and assuming I still had access to it and I could assure that it was the image that I made, that it somehow would itself be a problem. There are so many other ways this can back-fire I'm not going to try to enumerate them. I really hope no one put you in charge of operating anything critical...