9 ms·
Multiple airlines disrupted due to Microsoft Azure outage
- crazytony 2y agoThis rollercoaster is not over yet. There's a crowdstrike issue causing windows machines/servers to brick globally and this industry is heavily windows dependent. It may or may not be related to the Azure issue but it's suspicious to me. https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_error_in_latest_crowdstrike_update/ https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_e...
- Taniwha 2y agoHN discussion here - needs to be on the front page - here in NZ ATMs, supermarkets, satellite TV channels are all down https://news.ycombinator.com/item?id=41002195 https://news.ycombinator.com/item?id=41002195
- caf 2y agoYeah, it's far from just airlines affected. https://www.abc.net.au/news/2024-07-19/global-it-outage-crowdstrike-microsoft-banks-airlines-australia/104119960 https://www.abc.net.au/news/2024-07-19/global-it-outage-crow...
- mirekrusin 2y agoEurope will wake up to flood of problems as well. This needs to be at the top of HN. We are experiencing multiple issues here in EU. This issue feels extremely widely spread. Maybe we don't hear ppl complaining because we're offline? :)
- javaunsafe2019 2y agoBerlin airport is down
- nicbou 2y agoA few others as well, it seems.
- mitjam 2y agoJust in time for the beginning of school holidays in Berlin. Today, they expected 86.000 travelers in BER alone (https://www.airliners.de/flughafen-ber-erwartet-rekordzahl-passagieren-sommerferien/75506 https://www.airliners.de/flughafen-ber-erwartet-rekordzahl-p...)
- vesinisa 2y agoMaybe they have rolled back the update and Windows boxes in Europe are no longer pulling it?
- vladvasiliu 2y agoThey did around 8:30 CEST (6:30 GTM) as I understand it. Some of our servers managed to unbork themselves after a number of boot loops, but not all.
- surfingdino 2y agoRyanair is unable to check in passengers online. You can check in at the airport.
- ben_w 2y agoI had to physically stand in a queue for about 8 hours for a Ryanair customer support desk in an airport when the airport runway was closed by 1-2cm of snow. I forget the exact timing and can't be bothered to look up my notes, but it was something like 11pm to 7am at the origin airport for a flight that was supposed to have landed at the destination around 8pm, as we were also stuck on the runway for an hour or so and even getting that far had been delayed. The replacement flight the next day was also cancelled even though the airport was open. I ended up taking a ferry and a train, and that was still simultaneously faster than the next available Ryanair replacement flight and cheaper than any other provider on short notice. Fortunately I had an understanding boss who didn't mind me arriving 4 days later than expected, and also a place to crash for free while working out the best route home.
- geek_at 2y agoit's already noon in central europe and yes everythings fucked. except for the linux powered companies
- deleted 2y ago[deleted]
- dagaci 2y agoOn the wireless they are reporting a bad Crowdstrike update and a major Azure failover in central USA as separate events, are they they the same or different?
- SSLy 2y agoThey seem either unrelated, or the Azure one was caused by CrowdStrike.
- dagaci 2y agoA whole lot of people are running Crowdstrike in the cloud and on local PC An crowdstrike update last night caused a windows kernel panic Azure/Crowdstrike personel have spend rolled back the update in the cloud Local IT people will have to revert it from local machines manually
- bun_terminator 2y agoI have never heard of crowdstrike. Is that some kind of antivirus? How is that related to PCs not booting? And why does it affect so many PCs if I've never heard of it? I'm so confused
- shadow28 2y agoThey make security software that is really popular in various industries.
- davidmurdoch 2y agoThey make malware that steals funds from corporations (willingly!) so these corps can tick a security checkbox for some certification investors have been told is paramount; it's just disguised as security software.
- account42 2y agoSo in other words, they make "security" software just like any other security software company.
- moogly 2y agoBaffling name. Sounds more apt for a DDoS service.
- shrikant 2y agoIt's enterprise anti-malware that [in addition to other bits] has a client component installed on all PCs in the corporate network. An update to that client component (called an "endpoint") is causing those Windows machines to BSOD. It's unlikely you'd have heard of it unless you've worked at a large enterprise that runs primarily Microsoft IT. Crowdstrike does have Mac/Linux "endpoints" also (IIRC) but I'm unsure if they're affected as well.
- bun_terminator 2y agoI've never seen a non-Windows machine tbh. But our IT just send out an update that we don't use crowdstrike. Strange that I never heard of it if it's so widespread. But thanks
- maeil 2y agoAlmost certainly Azure using Crowdstrike on Windows in one way or another. Not surprising that AWS and GCP don't seem to be hit as they wouldn't run anything on Windows, unlike Azure, who I'm sure are forced to do so under MS' infamous interdepartmental structure.
- raverbashing 2y agoUgh. Though I can image there's a Azure market for "Citrix server" kinda thing in the cloud (or maybe it's SaaS - Solitaire as a Service)
- classified 2y agoCompanies using Windoze for anything touching customer business should get sued by their customers.
- craxdevil7 2y agotrue, got some insights why this happened https://medium.com/@confusedcyberwarrior/when-security-becomes-the-threat-the-crowdstrike-incident-9bbaeab9db9d https://medium.com/@confusedcyberwarrior/when-security-becom..., but how they didn't had an update process like testing or QA?
- protosam 2y agoThe article is pay walled. Seems like this would be the fault of the airlines though. There is a reason to be distributed between different geographic areas.
- tanelpoder 2y agoBut if the Azure outage is due to Windows machines crashing because of the currently ongoing CrowdStrike crash/reboot loop issue, then such servers might end up being down in all regions. Looks like there might be some advanced lessons to be learned about blast radius here...
- switch007 2y agoAzure VM host machines are running CrowdStrike...??
- bugbuddy 2y agoMaybe because Windows Defender Advanced Threat Protection is an enormous resource hog that scans every byte of memory and storage accessed by the Hypervisor and performs a quadratic time computation on the data? I am just guessing because my “fastest” Windows laptop CPU money could buy feels like a hot smelting furnace and a sloth at the same time when I use VMWare Workstation. What the &$@* is it scanning the VMWare guests for?
- pjc50 2y agoOh, Crowdstrike is also a massive resource hog.
- tanelpoder 2y agoProbably not, but maybe they rely on some Active Directory server that is running it?
- jaggederest 2y ago
- panarky 2y agohttps://archive.ph/FByL6 https://archive.ph/FByL6
- FearNotDaniel 2y agoA side issue, but since we’re on the subject of global tech being generally fscked: I’m currently on holiday in Italy and just discovered the entire archive.ph domain is blocked by the government, apparently due to kiddie porn. Shrug emoji…
- amarcheschi 2y agoI'm in italy and I can access archive.ph Perhaps just the ISP you're roaming with blocks the site?
- mr_world 2y agoThere was a law passed that allows ISPs to block sites that host copyrighted content illegally. It's not just Italy, also the Netherlands and many other countries. You can still access it with a vpn or tor. Tor browser on the phone works fine for mobile carriers that block the sites.
- giucal 2y agoYou don't even need a VPN. Just replace ISP's DNS servers with, say, Cloudflare's 1.1.1.1 1.0.0.1 or maybe OpenDNS's. [Edit: fixed wrong fallback address.]
- FearNotDaniel 2y agoYes that seems to be the case - the blocking page is headed with "Ministero dell'Interno - Dipartimento della Pubblica Sicurezza - Direzione Centrale per la Polizia Scientifica e la Sicurezza Cibernetica - Servizio Polizia Postale e Sicurezza Cibernetica" along with official looking government logos but text underneath (in Italian and English) talks about a collaboration between govt agencies and ISPs. From mobile cell service it was not blocked. The text does however mention it's a specific measure against child pornography, not re-hosting copyright content.
- kachinga123 2y agoNo flights at Germanys Berlin BER airport as well
- lifestyleguru 2y agoI bet BER staff is screaming at the passengers right now "you wanted this, it's your own fault".
- kolme 2y agoOh, you know them as well.
- LeoPanthera 2y agoBBC live coverage, no paywall. https://www.bbc.com/news/live/cnk4jdwp49et https://www.bbc.com/news/live/cnk4jdwp49et
- deleted 2y ago[deleted]
- switch007 2y agoBetter described as a worldwide IT outage https://www.bbc.com/news/live/cnk4jdwp49et https://www.bbc.com/news/live/cnk4jdwp49et Sky News UK is off the air. Some UK train companies are having an IT outage. Berlin, Melbourne airports flights disrupted...
- vergessenmir 2y agoWaitrose tills and card machines aren't working
- switch007 2y agoOK now the outage is getting real. How will Tarquin get his cinnamon and gooseberry yoghurt?
- openopenopen 2y agoDoes Microsoft use canary deployments? or was it deployed on everything?
- pplante 2y agoSounds like it's more of a problem caused by Crowdstrike. I'm sure azure fell over when millions of servers all freaked out simultaneously.
- jaggederest 2y agoGood lesson in rollout and heterogenous software stacks.
- eecc 2y agoI guess a canary rollout of the Crowdstrike patch would have contained the impact of this.
- BSDobelix 2y agoI love it! Just keep putting all your eggs in one basket, because at the end of the day, it's not your fault, but Azure's.
- switch007 2y agoI've had large customers seriously see that as a big appeal, when we were selling them some IT. They loved being able to blame someone else!
- tormeh 2y agoThe corporate training always says you are responsible for your chosen cloud vendor's problems, but in reality everyone assumes they can successfully blame the vendor.
- SteveSmith16384 2y ago"No-one ever got fired for choosing Microsoft".
- undebuggable 2y agoAlthough many got locked out for choosing them.
- BSDobelix 2y agoWell, the older version is about IBM, and it would probably even be true (today) if we were talking about mainframes, because they are one hell of a stable basket ;)
- tkubacki 2y agoThat’s why it’s important to demonopolize desktop OS market. Microsoft should be divided long time ago
- arcxi 2y agonobody is forcing these companies to depend on MS products and services
- deleted 2y ago[deleted]
- logicchains 2y agoMany in regulated industries actually are forced to use crapware like Crowdstrike, to tick a box on the security checkbox.
- arcxi 2y agoif they'd used crowdstrike on linux they wouldn't be hit with BSOD bootloops now
- YeahThisIsMe 2y agoHow would that change anything about Windows market share?
- switch007 2y agoYou don't think the bundling of windows with pcs has anything to do with their market share?
- commercialnix 2y agoSee my comment here https://news.ycombinator.com/item?id=41003860 https://news.ycombinator.com/item?id=41003860
- cm2187 2y agoTo those who remember the Y2K bug scare, Microsoft delivered it: https://www.youtube.com/watch?v=WhF7dQl4Ico https://www.youtube.com/watch?v=WhF7dQl4Ico
- dools 2y agoI'm so old that I was like "Do that many people really play Crowdstrike?" then I realised that's Counter Strike, then I looked up Counter Strike and it came out 24 years ago. yells at cloud
- robertlagrant 2y agoAnd yes, they do! Counterstrike 2 is out now, to a mixed reception.
- ClassyJacket 2y agoWhich is actually Counter Strike 4
- _the_inflator 2y agoThe dawn of the hybrid model: on premise will be back soon. The impact is vast: imagine being blocked by an hostile administration in this way. Disaster recovery of this magnitude is like a global pandemic. AI won’t save us. Network topology and admins will have their comeback.
- seydor 2y agoUnless attacks take down infrastructure regularly, we won't go back to decentralized model. The internet itself was created decentralized to withstand a a war .
- KineticLensman 2y agoNo, the ‘internet’ was created to allow US defence researchers to access shared computers more easily. It adopted the packet switched model that had been developed theoretically to support command and control applications, but which was never actually implemented by the extant C2 providers (in a dentist’s waiting room now so don’t have links)
- DanielHB 2y agoI dunno, I have been around and I never seen an on-prem infra being more reliable than your average cloud. The only difference is that when on-prem goes down you can shout to your infra engineers, when cloud goes down you shout at your enterprise cloud representative. The first is more effective, but even with that it still doesn't achieve the same reliability and disaster-recovery of your average cloud provider.
- rcxdude 2y agoIt also has the property of being less correlated with other failures, which may or may not be an advantage.
- tankenmate 2y agoWith small events cloud engineers scale better, with large events local engineers scale better.
- kaptainscarlet 2y agoThis is why I sometimes think on prem hosting is better.
- deleted 2y ago[deleted]
- belter 2y agoIt's 2024. Using Azure should be a firing offense.... https://www.lastweekinaws.com/blog/azures_vulnerabilities_are_quack/ https://www.lastweekinaws.com/blog/azures_vulnerabilities_ar...
- lifestyleguru 2y agoObviously you've never been working at enterprise purchasing department and participating in business meetings with Microsoft sales people.
- benterix 2y agoI see no relationship between understanding that Azure has a poor security posture and participating in business meetings with salespeople.
- lifestyleguru 2y agoThen you haven't met salespeople with high enough budget yet.
- totallywrong 2y agoYeah, you're completely out of touch of the industry. As horrendous as it is Windows on Azure (and the entire ecosystem it comes with) is an easy sell for the dinosaur IT leaders that haven't seen a Linux server in their life. You'd be surprised how common that is.
- javaunsafe2019 2y agoBerlin airport is down. I guess it’s related
- davidmurdoch 2y agoWaiting in a queue at an airport in Palma, Mallorca, Spain right now and the check in staff are currently flipping through printed sheets of paper to check us all in. It's going to be a very long wait.
- robertlagrant 2y agoDon't buy tills that run Windows. You must be crazy.
- steve1977 2y agoI wonder if these airlines were really affected by that Azure problem or if they were affected by the CrowdStrike issue and were just mixing things up.
- adamsiem 2y agoALB airport down. Check in systems are in BSOD ‘Recovery Mode’.
- surfingdino 2y agoSo, anti-malware software turns out to be malware? How odd. SecOps out on anti-open source training, I presume?
- totallywrong 2y agoAntivirus company causing exponentially more harm in hours than it's prevented during its entire existence.
- totallywrong 2y agoInteresting that I happened to transit 3 different Asian airports today and had zero issues. I haven't seen anything at all related to the outage over here.
- angra_mainyu 2y agoHmm, maybe they've migrated their systems? I know SK had a plan to migrate govt machines to Linux.
- totallywrong 2y agoHow is this update released at all? Something that affects literally every machine does not get caught in testing? From a 80B company no less.
- adultSwim 2y agoReal life operations of hospitals were affected (eg scheduled surgeries not taking place).