6 ms·
Microsoft says Delta's ancient IT explains long outage after CrowdStrike snafu
- Moto7451 2y agoMy wife worked at Travelport in the late 2010s. They ran a lot of Delta’s systems but Delta of course has their own infrastructure. Some of the history of Delta’s teams is intertwined with the companies that were merged together to form Travelport. From what I could tell from her talking about work it wasn’t particularly far off the state of the art at the time. They had a fancy new AWS hire to help lead them into AWS from mainframes. That didn’t fully work out but they still had a bunch of containerized .Net and Java running. Mostly this whole back and forth argument sounds like a ball of pain with everyone upset at each other and very little of it mattering unless lawsuits are filed. If Delta is using Windows at IBM it doesn’t make them “ancient.”
- 38 2y ago> containerized .Net and Java Gross. If that's the alternative I would rather stick with mainframes. People forget that it's perfectly fine and even better to just run code directly in many cases.
- Yasuraka 2y agoContainers are not VMs, code runs directly on metal and performance is pretty-much-native. https://web.archive.org/web/20150218172511/http://domino.research.ibm.com:80/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E7B/$File/rc25482.pdf https://web.archive.org/web/20150218172511/http://domino.res... This is from 2015, and it improved from there (e.g. the daemon-less podman matured, NAT is no longer needed). That said, I'd happily take even the performance impact of VMs and order a few more servers, if it means that I won't have to deal with dotnet/java runtimes being shared by multiple applications.
- 38 2y agoIf you need a container, your stack (dotnet/java) sucks. That's why Amazon deprecated the Go container, cause it's just not needed
- consteval 2y agoYes those langs have large runtimes. But even for compiled native languages, if you have a lot of dependencies you will require containers sooner or later. Go doesn't really have the huge dynamic-linking ecosystem. But, if you do and then project X and project Y diverge on versions... you have a problem. Also containers can automate deployment easier. Like the line from bare server -> full deployment is much faster if you use containers.
- jmount 2y agoSuffer a supplier failure, and your supplier is motivated to prove you are at fault.
- fortran77 2y agoMicrosoft didn't supply, recommend, or specify CrowdStrike.
- jmount 2y agoThey did supply Windows, and Microsoft is the one throwing shade.
- globalnode 2y agofrom what ive read microsoft is partially liable for allowing crowdstrike to make updates the way they did, forcing updates and then not detecting and disabling the affected driver. maybe im wrong. seems like theyre finger pointing and that makes me think theyre guilty even more.
- dvratil 2y agoWindows did not disable the driver because Crowdstrike marked their driver as critical for boot. Had they not done that, Windows would disable the driver automatically.
- senectus1 2y agodidn't you hear? the EU forced them to let Crowdstrike to be able to do it... This is the EU's fault :-P
- fortran77 2y agoEvery PC we have has IPMI. We can re-image any PC even--if it can't boot--remotely. Even if we don't have the bitlocker key, etc.
- lmm 2y agoWhat's your plan for if an attacker gets access to your IPMI system and uses it to put rootkits on all your machines, including the ones that you image other machines from?
- mschuster91 2y ago... or when that system gets hacked, like Mobile Guardian, where the hackers remote-wiped thousands of systems [1]. [1] https://www.heise.de/news/Mobile-Device-Management-Einbrecher-bei-Mobile-Guardian-loeschen-tausende-Geraete-9826415.html https://www.heise.de/news/Mobile-Device-Management-Einbreche...
- fortran77 2y agoThe IPMI network is airgapped
- asynchronous 2y agoIPMI?
- ainonsense44 2y agohttps://en.m.wikipedia.org/wiki/Intelligent_Platform_Management_Interface https://en.m.wikipedia.org/wiki/Intelligent_Platform_Managem...
- fortran77 2y agohttps://en.m.wikipedia.org/wiki/Intelligent_Platform_Management_Interface https://en.m.wikipedia.org/wiki/Intelligent_Platform_Managem...
- gizmo686 2y agoIntelligent Platform Management Interface Most severs have a management microcontroller in them that is capable of administoring the main system, typically including (but not limited to) full keyboard video and mouse access. Typically this is exposed through a dedicated ethernet port, allowing administrators to remotely manage their servers.
- throwaway290232 2y ago> CrowdStrike followed up with Delta on the offer for onsite support and was told that the onsite resources were not needed > Even though Microsoft's software had not caused the CrowdStrike incident, Microsoft immediately jumped in and offered to assist Delta at no charge following the July 19 outage," the letter said. "Each day that followed from July 19 through July 23, Microsoft employees repeated their offers to help Delta. Each time, Delta turned down Microsoft's offers to help, even though Microsoft would not have charged Delta for this assistance Two vendors offer you free on-site help, during the biggest outage in history, and you turn it down? This isn't the CIA, it's Delta Airlines. Bring those overpriced security monkeys in, make em sign an NDA, and have them sit there and watch you type. To not do that... even if they were useless... I can't find a justification ....... other than being able to later claim it wasn't Delta's fault it was down so long, and nobody having evidence to the contrary.
- anothername12 2y agoI think it’s absolutely bonkers that CrowdStrike software is basically an RCE
- lmm 2y ago"Security" software has always been like that. What's bonkers is the state of regulations and auditing where putting this kind of crap on your systems is somehow seen as increasing your security rather than decreasing it.
- theshrike79 2y agoThe guard watching your building needs to have full access to every room so that they can check if a bad guy got there without a key. You need to trust the guard implicitly.
- surfingdino 2y agoThe larger the org the slower they are with updates. Some global companies skip one or two versions of Windows because of the time it takes them to upgrade their systems. Microsoft is trying to deflect the blame, but it is their policy of maintaining backward compatibility that bit them. If you tell enterprise they can run their software on the latest version of your OS there is no incentive to upgrade, because they can always do it later. Steve Jobs knew how enterprise clients think from his time at NeXT and killed Apple's enterprise division when he came back. When large customers pushed him to create enterprise sales and support team he told them to go the nearest Apple Store, because he did not want them to limit his ability to change the OS.
- pjmlp 2y agoWell, that is why Apple is meaningless on the enterprise server room, and all their attempts at it flopped.
- kortilla 2y agoThat’s not why. It was overpriced and didn’t have a lot of software. Linux distros make changes and they ate Microsoft’s lunch in the server room.
- DaSHacka 2y ago> Linux distros make changes and they ate Microsoft’s lunch in the server room. This statement is a little funny considering the only reason the CrowdStrike situation was such a big deal was because of how many high-profile companies are using Windows Server.
- dlt713705 2y agoYou wouldn't want to live to see the day when the same situation happens on Linux servers.
- 2y ago
- mrweasel 2y agoThe article doesn't really explain anything to an extend that would all us to conclude anything regarding the age of those systems. The comment "because it runs on those providers' systems" is also a bit weird. Do they run on AS/400? Or is it simply because IBM would let Microsoft employees touch any IBM owned and operated infrastructure, probably the later. In my experience dealing getting access to infrastructure run by IBM (formerly SoftLayer), then yes, things might be ancient. We required VM with Windows XP to even run their VPN solution and that barely worked anyway.
- bsder 2y ago"Ancient" would be fine. At this point, we're pretty much back to a 3270 displaying data from a mainframe anyway. Use DOS and an 8514/A graphics card to access the "cloud" and we'll be better, faster and more secure than anything Javascript based. What is NOT fine is the lack of a disaster recovery plan. Your disaster recovery plan should be able to cold start yourself within 24 to 48 hours. It's clear that Delta simply didn't have anything resembling a disaster recovery plan.
- nomercy400 2y agoCan anybody explain why an IBM server can be affected by a Microsoft Windows crash? Why did the IBM server need restoring from backup? Also, wouldn't that contractually be IBM's issue to solve?
- coreyh14444 2y agoPure speculation: The part where the Delta team didn't want Microsoft's help feels like it is probably that they knew that their systems were a mess of old, unpatched Windows clients and servers and were embarrassed about it (and that their legal teams probably knew that it was better to keep everyone out so they could sue later on)
- ssahoo 2y agoWell, they dint buy their Azure windows VDI for help desk and flight status displays.
- jmclnx 2y ago>was being serviced by other technology providers, such as IBM, because it runs on those providers' systems, and not Microsoft Windows or Azure Another cute quote by M/S. I hope this goes to trial with a Jury. Then the truth will come out. Crowdfare crashed only Windows Systems, not IBM Mainframes or AS/400 or AIX or Linux or BSD. The blame is on Crowdfare and Windows only, not IBM systems.
- camilla4151 2y ago[dead]