12 ms·

Why the CrowdStrike bug hit banks hard

164 points by sideway 2y ago

pantulis 2y ago
This is a good writeup, but to be fair it's just not a matter of banking regulations. Basically all big companies are under similar obligations regarding endpoint protection.
- candiddevmike 2y ago
  Should endpoint protection require kernel level access? At what point does it stop becoming protection and start becoming a liability? Obligatory who watches/protects the watchmen/protector...
  - greenn 2y ago
    With the current model kernel level access is required. Real security products have to be able to operate above userland. Ideally in the future there can be a layer in between userland and kernel for this sort of thing. Maybe we use some of those extra protection rings?
    - ghusto 2y ago
      Couldn't you just ask some OS APIs provided by something in kernelspace for what you need? In fact, isn't this how macOS does things?
      - mywittyname 2y ago
        Microsoft was on their way to doing this, but was shot down by EU regulators because the APIs weren't available to all third-party vendors.
        armada651 2y ago
        I think it's kind of ridiculous to then blame the regulators for the fact that Microsoft decided not to go ahead with a more competitor-friendly design. The fact that Microsoft abandoned it as soon as a regulator pointed out how anti-competitive the design of the API was makes you wonder what Microsoft's true intention was. To me that implies the anti-competitive design was its main feature and to Microsoft it would've been pointless to continue without it.
        mywittyname 2y ago
        Maybe. Not working at MS I can't say what their reasons were. But another way of looking at this would be that perhaps they wanted to be the beta testers of the API themselves because opening it up would have been a maintenance liability for the company. Microsoft tends to be pretty good about backwards compatibility in ways that Apple is not. We also don't know that these APIs were cancelled, they may make it into future versions of windows.
      - btilly 2y ago
        You could, and in fact this is what Microsoft wanted to do. The EU said that they couldn't. And the reason why not is simple. Anything that Microsoft thinks is a good thing to add to the API, they'll add for themselves. When the new API is released, their software is released with it. This gives them a competitive advantage over competitors who have to wait for Microsoft to have the idea that they want, and then scramble to implement it after Microsoft does. The EU is suspicious of this for the simple reason that Microsoft has a several decade history of doing exactly that. Repeatedly. My favorite example being the release of Windows 95 with Microsoft Word available at the same time, and with WordPerfect unable to run. By the time WordPerfect had figured out how to port their software to Windows 95, they were no longer the market leader.
        ghusto 2y ago
        Interesting! I guess there's no way to fix this with further regulation either, since it would be some work to prove MS had access to the API contracts before they released them. The ultimate lesson then is to stop using MS stuff.
        lucianbr 2y ago
        The way I see it, Microsoft sells some antivirus software, and also gets to decide who is allowed or not to compete with their antivirus software, by providing or denying access to the API. Obviously unfair.
        bluGill 2y ago
        I think anti-virus should be part of the core os. This does kill all third party vendors - good riddance to most of them, sorry if there is one that isn't evil (I'm not aware of it)
        lucianbr 2y ago
        Once the AV vendors exist, killing them, especially by Microsoft, is clearly anticompetitive. If you could prevail on a government to decide that, maybe it could work. One thing I see, is that AV has a component of maintaining a DB of signatures of bad things. This does not seem at all the job of the core os. Would the Debian team maintain such a DB?
    - jen20 2y ago
      > With the current model kernel level access is required. On Windows.
      - c0balt 2y ago
        Note: At least on Linux the main alternatives for this, either eBPF (e.g., pulsar or falcon) or a kernel module, both require this too.
        kelnos 2y ago
        eBPF is at least somewhat sandboxed, no? So it doesn't quite have the access required to accidentally stomp on any portion of kernel memory it wants?
        c0balt 2y ago
        Indeed it's executed via a Jit on something like a VM. However it can still, make your system quite disfunctional if, e.g., all filesystem or network calls are blocked.
        vel0city 2y ago
        The version of the CrowdStrike sensor that caused kernel panics on RHEL/Rocky was using eBPF. It living in eBPF doesn't mean it can't cause system instability. And as mentioned elsewhere, an eBPF module behaving badly but in valid ways can still make your system pretty unusable.
        jen20 2y ago
        macOS does not require this however.
    - tmm 2y ago
      > Maybe we use some of those extra protection rings? Maybe not. Intel is considering removing rings 1 and 2 for a future 64-bit only x86 architecture, because they "are unused by modern software". https://www.intel.com/content/www/us/en/developer/articles/technical/envisioning-future-simplified-architecture.html https://www.intel.com/content/www/us/en/developer/articles/t...
      - bluGill 2y ago
        I don't think those extra rings would be useful for what is needed anyway.
  - jabroni_salad 2y ago
    If you don't do it, someone else will. Unless the OS is locked down to the point that even its owner cannot do that. Actually, this is something I like about Operational Technology, you run into a lot of doodads where the elevation process requires turning a physical key, and the device's main functionality is disabled while it is in service mode. Ofc the doodad has to be engineered to operate reliably, perpetually, for years, and you cant really expect that from a desktop computer.
    - bluGill 2y ago
      I have said for 20 years now that Microsoft Word should have a check on startup, if the current user is administrator it should put up a message that administrators are not allowed to use a Word Process, login as someone else. This one change would solve a lot of problems. Even on home machines where no user has a password, having to do something special to get into administrator mode will stop several attacks just because people will slow down and ask.
      - codewench 2y ago
        That's pretty much what Microsoft tried with the UAC prompts, and that was fairly universally disliked. Not that I disagree with you, running as admin by default is a terrible practice, but it's a tough sell to the general public
      - Dalewyn 2y ago
        Administrators can and should be able to do anything and everything, that is literally an administrator's job description. Also, if you want to stop everyone from using administrator accounts, the simplest way is to not have the Windows installer/OOBE setup make an administrator account first. Windows has a built-in Administrator account already not unlike Root in Linux, there is no reason (other than tradition and absolute convenience) the Windows installer/OOBE setup needs to make an administrator account for the user installing/setting up.
      - Vogtinator 2y ago
        Would that actually have a positive effect? Running malicious software in the only user's context can already cause maximum damage: https://xkcd.com/1200/ https://xkcd.com/1200/ This would just result in more UAC prompts and thus annoyed users who get taught to click on "Allow" whenever a dialog pops up.
  - adrr 2y ago
    How else would you monitor a windows box? EU won't allow Microsoft to lock down their kernel and provide MacOS type solution with APIs for trust publishers.
    - imtringued 2y ago
      From what I have heard, Microsoft is allowed to do that, they merely aren't allowed to be a competitor to the software that uses the API.
  - supriyo-biswas 2y ago
    > At what point does it stop becoming protection and start becoming a liability? If such outages were more frequent, then it could definitely become a liability. But such risks have to be balanced against the risk of being compromised and leaking customer data and other confidential trade secrets, and the risk posed by the latter one is far higher, not to say it's also more common.
  - SkyPuncher 2y ago
    Yes. Absolutely yes. It's the only way to detect certain types of advanced threats.
  - notepad0x90 2y ago
    Yes, it needs kernel access given the userspace api's available in windows. Period. not a single person who knows how the tool works and the threats it protects against has said other wise. userpace can't disable or tamper kernel space but an admin/root process in userspace can.
    - candiddevmike 2y ago
      FWIW I asked if it should require access, not what the current status quo is/what limitations exist within the OS.
      - notepad0x90 2y ago
        It isn't a status quo, it is the design of the windows operating system, as well as Linux. Macos does it's own thing but it is somewhat effective because you need to go into recovery before you can disable sysexts as root. Imagine needing to go to windows recovery environment to disable drivers, that won't fly. Apple can do that because they control the hardware and software, you rarely need to mess with sysexts as part of troubleshooting as a result. Unlike normal software development, anti-malware software has to be resilient against all kinds of tampering. The price for having an os that isn't heavily locked down and tamper resistant due to hardware enabled checks is having to rely on kernel mode code to enforce tamper resistance. Evasion is another issue, you can already hook api calls from user space (some EDRs do this) but evading it as a privileged user is trivial. It boils down to how on x86/x64 the cpu enforces 3 major privilege rings, by design things that are integrated with the OS that require OS level privileges and system wide access must run in the same ring as the OS (ring0/kernel mode). There are many ways to tackle this but I haven't heard of any (even from Microsoft's blogs/proposals after the incident) that won't reduce the capabilities and tamper/evasion resiliency of these security softwares. if x64 had a "secure world" concept like ARM for example, that would be different but it doesn't.
- SoftTalker 2y ago
  If not regulations, then demands by insurers for cyberattack insurance coverage.
- SkyPuncher 2y ago
  Basically all B2B companies are under some sort of obligation to have endpoint protection. All of these requirements essentially become transitive across a company's entire supply chain. * Big bank needs to comply with X, so do all of their vendors. * Vendor wants to sell to big bank, so they comply with X. They also need all of their vendors to comply with X. * So on and so on. ---- Ultimately, there are a lot more options than CrowdStrike, but this is a case of "Nobody gets fired for buying IBM". Even if CrowdStrike isn't the "best", it's good enough. Because it's use is sooo widespread, an issue with it often affects dozens and dozens of other companies when you're affected. One of the great things about this effect is everyone "goes down at the same time", so people don't tend to point fingers at you. In fact, they might not have any clue you're down because some other, more critical system is down internally and preventing them from accessing you. I remember a similiar situation happening a few years back. A big outage hit large parts of the internet. A pretty major part of our app got taken offline with this outage. This was a known risk and something that we accepted. We expected some backlash and inquires if this situation should ever happen. It was a calculated risk to dedicate more effort towards building customer-facing value. I think we got one inquiry. It was basically just an FYI. This person had so many things broken on their end that "one more thing" being broken was just a drop in the bucket.
  - pantulis 2y ago
    Yes, this is a good summary of the situation. As a matter of fact, I guess there were quite a lot of systems and services that went down even though they were not using Crowdstrike themselves, but some part of their cloud supply chain was. I see Salesforce and Adobe were impacted in some way, probably due to the collateral Azure disruption. On the other hand, count me surprised at the sales prowess of Crowdstrike, I did not know how big they were.
deepsun 2y ago
I'm still amazed how the blame shifted from Microsoft to CrowdStrike. Yes, CrowdStrike update caused that -- but applications fail all the time. It was Microsoft's oversight to put it on Windows critical path. And banks/airlines etc were hit hard because their _Windows_ didn't boot, not because of an application crash on a perfectly working Windows.
- hulitu 2y ago
  I think they said it was a windows driver, not a normal application. Running crap in kernel mode does not end well on any OS.
  - concerned_user 2y ago
    Yes it is a driver which is signed and tested by Microsoft. Driver allows to run arbitrary unsigned code. Why is that allowed?
    - cyberpunk 2y ago
      The driver is some kind of AV/Signature detection hook. E.g check every open() for this list of checksums and refuse to open known viruses style system. The 'update' was a borked definition file which triggered a bug in that system. It's not code execution without signing, and I think probably they do want these files to be updated hands free. The real problem was the lack of testing, rather than the actual mechanism I think.
    - Joker_vD 2y ago
      ...you want Microsoft to forbid you from running certain kinds of programs on your own machine, even if you really, really insist on it, do I understand you correctly?
      - hpen 2y ago
        More like: "...you want Microsoft to forbid you from running certain kinds of programs (with gaping security holes / processes) on your own machine" YES
btbuildem 2y ago
The takeaway from this article seems to be: buy crowdstrike shares, because major corps are unable to make any changes, and will continue to pay licensing fees for this "service" for the foreseeable future.
- deleted 2y ago
  [deleted]
- tootie 2y ago
  This is going to crush their sales pipeline and lead to at least a few attempting a migration off. Crowdstrike is unlikely to go out of business, but this is not a good time to buy.
  - nkassis 2y ago
    SolarWinds comes to mind they haven't fully recovered but they are still around and kicking.
  - alephnerd 2y ago
    Safe Harbor: Don't follow random internet commentators opinions on public markets. This is just an opinion and not advice. I disagree. Long term, the fundamentals of CRWD continue to remain unabated. Endpoint protection is still a critical need no matter what - for every bug like CRWD, there's always a company you can point to who's operations were shut down due to an attack. CRWD skimped on QA and customer support, but long term there aren't many other vendors that can provide a similar service, and CRWD is large enough to pull a PANW and M&A into entirely new segments (eg. DSPM with Flow Security, Observability/Data Lake with Humio, ASPM with Bionic) along with greenfield category makers like Charlotte AI for AI Security and AI EDR. There will be short term pain for CRWD's Windows endpoint business with churn to MDE, SentinelOne, Tanium, etc but they have enough dry powder and a diversified security portfolio that they can safely recover within a year at most. > crush their sales pipeline With CRWD sized companies, most of their revenue comes from multi-year contracts and renewals. They'll probably have a decently large layoff in the sales org, but enterprise sales tends to be fairly stable due to contract sizes along with riders about liability
Retr0id 2y ago
> For historical reasons, that area where almost everything executes is called “userspace.” It's an old term at this point, but I don't think the reasons for it being called "userspace" have changed or become outdated since then, so I wouldn't call them historic per se.
- deleted 2y ago
  [deleted]
- Macha 2y ago
  Things have gotten messier with virtualization, containerisation, hypervisors etc. The internet loves to produce pedants to argue the post should go into the finer points of these even when it's not relevant to the message. And so people like the author have a defensive reflex to throw in some language to bounce the pedants away.
- SoftTalker 2y ago
  I used to like Patrick's posts but lately they are way to long and full of irrelevant minutia. Decide who you're writing for, and write to that audience.
  - rescbr 2y ago
    Some of his audience likes the irrelevant minutia.
  - rozenmd 2y ago
    > Decide who you're writing for, and write to that audience. He has, and he does.
  - arduanika 2y ago
    Congrats, you've been screenshotted and tweeted by him! "In which an HN commenter offers me writing advice but fails to understand the implication of second sentence" https://x.com/patio11/status/1818757982706139297 https://x.com/patio11/status/1818757982706139297
- shadowgovt 2y ago
waihtis 2y ago
Regulations are a big reason why this happened, sure, but also it hit the companies with great security budgets more. Hospitals, for instance, weren't that widely affected as they barely have any money to buy security tooling. Silver linings and all that, I guess.
- cookiengineer 2y ago
  > Hospitals Everybody seems to be quick to forget about WannaCry.
  - c0balt 2y ago
    Wannacry was not an accident. It was inarguably an intentional attach against general IT infrastructure instead of a borked update.
  - waihtis 2y ago
    That really just proves my point.
toddmorey 2y ago
Was anyone else surprised how little disruption they personally experienced? I had braced for impact that weekend. But all my flights were perfectly on time, all my banking worked, providers worked, and sites & resources were available. I don’t know if I somehow just have little exposure to Windows in my life or if there’s an untold resiliency story for the global internet in the face of such a massive outage. All I can say is THANK YOU to all the unsung heroes who answered the call and worked their butts off. Infrastructure doesn’t work without you. We see you & we thank you!
- marcosdumay 2y ago
  People found a really quick workaround. It would take a couple more days to fix if there wasn't any.
- davio 2y ago
  I was unaffected on my work laptop. One of my coworkers is a long-timer and said when the company first got laptops there was a huge "OMG leave your laptops on overnight" push to make sure updates were applied. I always at least sleep if not shut-down after work so I guess I missed out
  - doubled112 2y ago
    I know at least one person who "survived" while her coworker's laptops were down. My first question was "do you shut your machine off at the end of the day?" She did, and that's probably why about half of her office was affected, and the other half was not. Can't update it if it isn't on.
- blackoil 2y ago
  IIRC only 5% of Windows machines were affected. So, it is very probable that most people just saw the news but have no real impact on them. Some had minor and maybe memorable impact, like Indian airlines giving handwritten boarding passes.
  - bostik 2y ago
hpen 2y ago
We blame car manufacturers for defects from suppliers, but we don't blame platform manufacturers (Microsoft) for holes in their architecture?
- SirMittens 2y ago
  I think that's the wrong analogy. A more correct one would be "Should we blame a car company for a broken engine, that was modified after it was sold to you?". A kernel level driver from a 3rd party is something that you willingly add to the OS, it wasn't there. Just because windows allow you to do it, doesn't mean you should. I mean, you can apply some dangerous mods to your car's engine, but you probably shouldn't, and if you do, it's your responsibility, not the car company.
  - hpen 2y ago
    Does crowdstrike void the warranty like an engine add on?
    - vel0city 2y ago
      If you had a support contract with Microsoft for your Windows installs and CrowdStrike is breaking your system they'll tell you to go talk to CrowdStrike, yes.
      - hpen 2y ago
        Ok I didn't realize that crowdstrike was more of competitor or maybe a hacky add-on (like a NOS). I was under the impression that it was something more in cooperation (not owned by or anything) but with Microsoft in terms of market support.
        vel0city 2y ago
        CrowdStrike absolutely is a competitor to Microsoft. Microsoft sells licensed software in the exact same market as CrowdStrike. Microsoft even sells Microsoft Defender for MacOS and Linux. They're direct competitors. https://www.microsoft.com/en-us/security/business/endpoint-security/microsoft-defender-endpoint?msockid=0fe5b142a836612518bda1dba9976023 https://www.microsoft.com/en-us/security/business/endpoint-s... https://learn.microsoft.com/en-us/defender-endpoint/non-windows https://learn.microsoft.com/en-us/defender-endpoint/non-wind...
deleted 2y ago
[deleted]
jmuguy 2y ago
Maybe the IT departments at the affected orgs take solace in the fact that so many other orgs had issues that the heat is off - but in my opinion this was still a failure of IT itself. There's no reason that update should have been pushed automatically to the entire fleet. If Crowdstrike's software doesn't give you a way to rollout updates on a portion of your network before the entire fleet, it shouldn't be used.
- candiddevmike 2y ago
  The update bypassed the controls orgs had in place to defer/schedule updates, AFAIK.
  - jmuguy 2y ago
    I've had trouble nailing down if thats the case from searching around online. And if thats true - thats absolutely on Crowdstrike. And that behavior should disqualify it from being used on critical systems. I imagine this incident will cause a lot of teams to consider just what can happen automatically on their systems.
    - tantalor 2y ago
      [flagged]
      - lucianbr 2y ago
        How did 911 services go down then? Whatever system caused that, should be by definition critical, imho.
      - Sohcahtoa82 2y ago
        You're living in a different reality. I can't fathom how anybody could legitimately make that claim. Even if you're defining "critical system" as "critical to humans" and not "critical to the business", then sure, you can say "Airlines aren't critical" and for most passengers, yeah, you're probably right. Most industries aren't critical, so businesses being ground to a halt doesn't matter for the consumers. But 911 systems were affected, and those are certainly critical to humans. If 911 doesn't work, ambulances and fire trucks can't be dispatched, and people die. EDIT: Computers attached to hospital beds, including trauma surgery rooms, were affected. I'm really curious what you think defines a critical system.
__MatrixMan__ 2y ago
I like the technical stuff here. I'm not so sure about this: > money is core societal infrastructure, like the power grid and transportation systems are. It would be really bad if hackers working for a foreign government could just turn off money. Sure, it would be inconvenient in the short term. But I think the current design is holding us back. I suspect that most of us would have more to gain than to lose if we managed to shut off money-as-we-know-it and keep it off for long enough to iterate on alternatives. Any design that even tried to step beyond "well that's how we've always done it" would likely land somewhere better than what we're doing. Much has changed since Alexander Hamilton.
- gadders 2y ago
  In the short term people would probably starve to death.
  - Joker_vD 2y ago
    Probably not. A competent government could install temporary rationing for the most essential goods such as food. It happened through the the whole of the 1917—1920 Russian revolution, with four or five kinds of paper money being circulated around, and the urban population managed through it only if barely. That government was much less competent than the US government is today.
    - mminer237 2y ago
      I mean, millions still starved during the revolution, even with the American Relief Administration feeding 10% of the country.
      - Joker_vD 2y ago
        In the rural areas, mind you. That's one of the most appalling thing about famines in the XIX-XX, that they hit the countryside heavier than they hit the cities.
    - deleted 2y ago
kristaps 2y ago
The article specifically mentions US banks and as I personally didn't see any disruption over here - is there (anec)data on how popular CrowdStrike is in the US vs the EU?
- Muromec 2y ago
  Can't have disruption from CrowdStrike if you run on IBM mainframes with cobol coz your math only opens gates for new technologies once in 25 years.
  - MattSayar 2y ago
    Oh wow an Anathem reference! To answer the question, CrowdStrike is a global company with thousands of employees around the world. Not sure why the EU wasn't hit as hard.
- Ekaros 2y ago
  Might be question what type of disruption it is. Transfers and web bank is likely to work. Branches offices and ATMs might have issues. So if you try to do anything in person or negotiate anything with workers in bank there could be issues.
adrr 2y ago
Did it really hit banks hard? Core banking systems don't run windows, they run on mainframes typically on IBM z/OS. I know it hit the financial firms hard and knocked out their trading systems but I don't know of any major bank losing their core bank system due to crowdstrike. Australia got hit hard because they modernized their bank systems and now most are cloud based. I am not aware of any major bank running their core systems on the cloud or on windows.
- tempodox 2y ago
  > they modernized their bank systems You mean they made them more vulnerable?
- josephthejoe 2y ago
  [dead]
taeric 2y ago
Any explanation that doesn't boil this down to "software required by corporate policy checklist not written by technical team" is almost certainly missing something here. This is almost definitionally policy capture by a security team and the all too common consequences that attach. The section that goes over why this wasn't federally pushed is largely accurate, mind. Not all capture is at the federal level. Is why you can get frustrated with customer support for asking you a checklist of unrelated questions to the problem you have called in. And the super frustrating thing is that these checklists are often very effective for why they exist.
voytec 2y ago
> Another way is if it has recently joined a botnet orchestrated from a geopolitical adversary of the United States after one of your junior programmers decided to install warez because the six figure annual salary was too little to fund their video game habit. Fictional statements like this make me reluctant to read further, and ignore source of such "news" in the future.
- davidgerard 2y ago
  what makes you think it was fictional? also, bragging about your inability to read text seems an odd way to interact.
  - voytec 2y ago
    Bragging? Reluctant==unable?
- bdamm 2y ago
  It's obviously fictional, but let's call it contemporary drama based on a true story. I thought the point was well made. The author already noted this was a handwaving segment.
- samspot 2y ago
  I got in trouble for something like this early in my career (running bittorrent over my work vpn).
bob1029 2y ago
I feel like this only impacted the larger banks. I've heard absolutely no explosion noises coming from smaller institutions. The effect of regulations and their enforcement is felt differently across the spectrum. There is something to be said for a diverse banking industry when it comes to this kind of problem. Also, this event is a powerful argument for keeping the core systems on unusual mainframe architectures. I think building a bank core on windows would be a really bad choice, but some vendors have already done this.
MadVikingGod 2y ago
While reading this I was struck with an interesting question: What risk does any particular software vendor pose to an industry at large? For example (making up numbers here): if 75% of all airline computers have croudstrike falcon installed that seems like a very concentrated risk. I actually wouldn't be surprised if we had this we would see really high concentrations of a small number of vendors in any industry.
- anticristi 2y ago
  The EU DORA regulation (Digital Operational Resilience Act for Financial Entities) has explicit provisions to avoid concentration risks. I heard a story that a bank was forced to use Google Cloud, because two other banks were already on AWS and Azure.
- saltminer 2y ago
  Alternatively, if Oracle hikes the price on an industry-specific product by 75%, how much of that industry goes under?
shadowgovt 2y ago
Just as a general comment on this whole affair: This would be the third incident I'm familiar with of a file of entirely zeroes breaking something big. Folks, as much as we wish it weren't true, null comes up all the damn time, and if you don't have tests trying to force-feed null into your system in novel and exciting ways, production will demonstrate them for you. Never assume 'zero' (for whatever form zero takes in context) can't be an input.
- tempodox 2y ago
  As long as the botchers get away with impunity, they won't “waste” resources on higher standards.
saltminer 2y ago
> This created a minor emergency for me, because it was an other-than-minor emergency for some contractors I was working with. > Many contractors are small businesses. Many small businesses are very thinly capitalized. Many employees of small businesses are extremely dependent on receiving compensation exactly on payday and not after it. And so, while many people in Chicago were basically unaffected on that Friday because their money kept working (on mobile apps, via Venmo/Cash App, via credit cards, etc), cash-dependent people got an enormous wrench thrown into their plans. I never really thought about not having to worry about cashflow problems as a privilege before, but it makes sense, considering having access to the banking system to begin with is a privilege. I remember my bank's website and app were offline, but card processing was unaffected - you could still swipe your cards at retailers. For me, the disruption was a minor annoyance since I couldn't check my balance, but I imagine many people were probably panicking about making rent and buying groceries while everything was playing out.
- HeyLaughingBoy 2y ago
  The really admirable thing about this is that Patrick acknowledged that it was "an other-than-minor emergency" for the contractors and took steps to ensure that they were paid rapidly. In a similar situation many people would have shrugged and taken an attitude of "sorry, bank's down. I'll pay you when it comes back up."
HeyLaughingBoy 2y ago
> Configuration bugs are a disturbingly large portion of engineering decisions which cause outages I work in medical device software -- the stuff that runs on machines in hospital labs, ER's or at patient bedside. The first "ohmigod do we need to recall this?" bug I remember was an innocuous piece of code that was inserted to debug a specific problem, but which was supposed to be disabled in the "non-debug" configuration. Then somehow, the software update shipped with a change to the configuration file that enabled that code to run. Timing-critical debug code running on a real-time system with a hard deadline is a recipe for disaster. Thankfully, we got out of that pretty easily before it affected more than a small handful of users, but things could have been a lot worse.

164 points by sideway 2y ago

This is a good writeup, but to be fair it's just not a matter of banking regulations. Basically all big companies are under similar obligations regarding endpoint protection.

candiddevmike 2y ago
Should endpoint protection require kernel level access? At what point does it stop becoming protection and start becoming a liability? Obligatory who watches/protects the watchmen/protector...
- greenn 2y ago
  With the current model kernel level access is required. Real security products have to be able to operate above userland. Ideally in the future there can be a layer in between userland and kernel for this sort of thing. Maybe we use some of those extra protection rings?
  - ghusto 2y ago
    Couldn't you just ask some OS APIs provided by something in kernelspace for what you need? In fact, isn't this how macOS does things?
    - mywittyname 2y ago
      Microsoft was on their way to doing this, but was shot down by EU regulators because the APIs weren't available to all third-party vendors.
      - armada651 2y ago
        I think it's kind of ridiculous to then blame the regulators for the fact that Microsoft decided not to go ahead with a more competitor-friendly design. The fact that Microsoft abandoned it as soon as a regulator pointed out how anti-competitive the design of the API was makes you wonder what Microsoft's true intention was. To me that implies the anti-competitive design was its main feature and to Microsoft it would've been pointless to continue without it.
        mywittyname 2y ago
        Maybe. Not working at MS I can't say what their reasons were. But another way of looking at this would be that perhaps they wanted to be the beta testers of the API themselves because opening it up would have been a maintenance liability for the company. Microsoft tends to be pretty good about backwards compatibility in ways that Apple is not. We also don't know that these APIs were cancelled, they may make it into future versions of windows.
    - btilly 2y ago
      You could, and in fact this is what Microsoft wanted to do. The EU said that they couldn't. And the reason why not is simple. Anything that Microsoft thinks is a good thing to add to the API, they'll add for themselves. When the new API is released, their software is released with it. This gives them a competitive advantage over competitors who have to wait for Microsoft to have the idea that they want, and then scramble to implement it after Microsoft does. The EU is suspicious of this for the simple reason that Microsoft has a several decade history of doing exactly that. Repeatedly. My favorite example being the release of Windows 95 with Microsoft Word available at the same time, and with WordPerfect unable to run. By the time WordPerfect had figured out how to port their software to Windows 95, they were no longer the market leader.
      - ghusto 2y ago
        Interesting! I guess there's no way to fix this with further regulation either, since it would be some work to prove MS had access to the API contracts before they released them. The ultimate lesson then is to stop using MS stuff.
      - lucianbr 2y ago
        The way I see it, Microsoft sells some antivirus software, and also gets to decide who is allowed or not to compete with their antivirus software, by providing or denying access to the API. Obviously unfair.
        bluGill 2y ago
        I think anti-virus should be part of the core os. This does kill all third party vendors - good riddance to most of them, sorry if there is one that isn't evil (I'm not aware of it)
        lucianbr 2y ago
        Once the AV vendors exist, killing them, especially by Microsoft, is clearly anticompetitive. If you could prevail on a government to decide that, maybe it could work. One thing I see, is that AV has a component of maintaining a DB of signatures of bad things. This does not seem at all the job of the core os. Would the Debian team maintain such a DB?
  - jen20 2y ago
    > With the current model kernel level access is required. On Windows.
    - c0balt 2y ago
      Note: At least on Linux the main alternatives for this, either eBPF (e.g., pulsar or falcon) or a kernel module, both require this too.
      - kelnos 2y ago
        eBPF is at least somewhat sandboxed, no? So it doesn't quite have the access required to accidentally stomp on any portion of kernel memory it wants?
        c0balt 2y ago
        Indeed it's executed via a Jit on something like a VM. However it can still, make your system quite disfunctional if, e.g., all filesystem or network calls are blocked.
        vel0city 2y ago
        The version of the CrowdStrike sensor that caused kernel panics on RHEL/Rocky was using eBPF. It living in eBPF doesn't mean it can't cause system instability. And as mentioned elsewhere, an eBPF module behaving badly but in valid ways can still make your system pretty unusable.
      - jen20 2y ago
        macOS does not require this however.
  - tmm 2y ago
    > Maybe we use some of those extra protection rings? Maybe not. Intel is considering removing rings 1 and 2 for a future 64-bit only x86 architecture, because they "are unused by modern software". https://www.intel.com/content/www/us/en/developer/articles/technical/envisioning-future-simplified-architecture.html https://www.intel.com/content/www/us/en/developer/articles/t...
    - bluGill 2y ago
      I don't think those extra rings would be useful for what is needed anyway.
- jabroni_salad 2y ago
  If you don't do it, someone else will. Unless the OS is locked down to the point that even its owner cannot do that. Actually, this is something I like about Operational Technology, you run into a lot of doodads where the elevation process requires turning a physical key, and the device's main functionality is disabled while it is in service mode. Ofc the doodad has to be engineered to operate reliably, perpetually, for years, and you cant really expect that from a desktop computer.
  - bluGill 2y ago
    I have said for 20 years now that Microsoft Word should have a check on startup, if the current user is administrator it should put up a message that administrators are not allowed to use a Word Process, login as someone else. This one change would solve a lot of problems. Even on home machines where no user has a password, having to do something special to get into administrator mode will stop several attacks just because people will slow down and ask.
    - codewench 2y ago
      That's pretty much what Microsoft tried with the UAC prompts, and that was fairly universally disliked. Not that I disagree with you, running as admin by default is a terrible practice, but it's a tough sell to the general public
    - Dalewyn 2y ago
      Administrators can and should be able to do anything and everything, that is literally an administrator's job description. Also, if you want to stop everyone from using administrator accounts, the simplest way is to not have the Windows installer/OOBE setup make an administrator account first. Windows has a built-in Administrator account already not unlike Root in Linux, there is no reason (other than tradition and absolute convenience) the Windows installer/OOBE setup needs to make an administrator account for the user installing/setting up.
    - Vogtinator 2y ago
      Would that actually have a positive effect? Running malicious software in the only user's context can already cause maximum damage: https://xkcd.com/1200/ https://xkcd.com/1200/ This would just result in more UAC prompts and thus annoyed users who get taught to click on "Allow" whenever a dialog pops up.
- adrr 2y ago
  How else would you monitor a windows box? EU won't allow Microsoft to lock down their kernel and provide MacOS type solution with APIs for trust publishers.
  - imtringued 2y ago
    From what I have heard, Microsoft is allowed to do that, they merely aren't allowed to be a competitor to the software that uses the API.
- supriyo-biswas 2y ago
  > At what point does it stop becoming protection and start becoming a liability? If such outages were more frequent, then it could definitely become a liability. But such risks have to be balanced against the risk of being compromised and leaking customer data and other confidential trade secrets, and the risk posed by the latter one is far higher, not to say it's also more common.
- SkyPuncher 2y ago
  Yes. Absolutely yes. It's the only way to detect certain types of advanced threats.
- notepad0x90 2y ago
  Yes, it needs kernel access given the userspace api's available in windows. Period. not a single person who knows how the tool works and the threats it protects against has said other wise. userpace can't disable or tamper kernel space but an admin/root process in userspace can.
  - candiddevmike 2y ago
    FWIW I asked if it should require access, not what the current status quo is/what limitations exist within the OS.
    - notepad0x90 2y ago
      It isn't a status quo, it is the design of the windows operating system, as well as Linux. Macos does it's own thing but it is somewhat effective because you need to go into recovery before you can disable sysexts as root. Imagine needing to go to windows recovery environment to disable drivers, that won't fly. Apple can do that because they control the hardware and software, you rarely need to mess with sysexts as part of troubleshooting as a result. Unlike normal software development, anti-malware software has to be resilient against all kinds of tampering. The price for having an os that isn't heavily locked down and tamper resistant due to hardware enabled checks is having to rely on kernel mode code to enforce tamper resistance. Evasion is another issue, you can already hook api calls from user space (some EDRs do this) but evading it as a privileged user is trivial. It boils down to how on x86/x64 the cpu enforces 3 major privilege rings, by design things that are integrated with the OS that require OS level privileges and system wide access must run in the same ring as the OS (ring0/kernel mode). There are many ways to tackle this but I haven't heard of any (even from Microsoft's blogs/proposals after the incident) that won't reduce the capabilities and tamper/evasion resiliency of these security softwares. if x64 had a "secure world" concept like ARM for example, that would be different but it doesn't.
SoftTalker 2y ago
If not regulations, then demands by insurers for cyberattack insurance coverage.
SkyPuncher 2y ago
Basically all B2B companies are under some sort of obligation to have endpoint protection. All of these requirements essentially become transitive across a company's entire supply chain. * Big bank needs to comply with X, so do all of their vendors. * Vendor wants to sell to big bank, so they comply with X. They also need all of their vendors to comply with X. * So on and so on. ---- Ultimately, there are a lot more options than CrowdStrike, but this is a case of "Nobody gets fired for buying IBM". Even if CrowdStrike isn't the "best", it's good enough. Because it's use is sooo widespread, an issue with it often affects dozens and dozens of other companies when you're affected. One of the great things about this effect is everyone "goes down at the same time", so people don't tend to point fingers at you. In fact, they might not have any clue you're down because some other, more critical system is down internally and preventing them from accessing you. I remember a similiar situation happening a few years back. A big outage hit large parts of the internet. A pretty major part of our app got taken offline with this outage. This was a known risk and something that we accepted. We expected some backlash and inquires if this situation should ever happen. It was a calculated risk to dedicate more effort towards building customer-facing value. I think we got one inquiry. It was basically just an FYI. This person had so many things broken on their end that "one more thing" being broken was just a drop in the bucket.
- pantulis 2y ago
  Yes, this is a good summary of the situation. As a matter of fact, I guess there were quite a lot of systems and services that went down even though they were not using Crowdstrike themselves, but some part of their cloud supply chain was. I see Salesforce and Adobe were impacted in some way, probably due to the collateral Azure disruption. On the other hand, count me surprised at the sales prowess of Crowdstrike, I did not know how big they were.

deepsun 2y ago

I'm still amazed how the blame shifted from Microsoft to CrowdStrike. Yes, CrowdStrike update caused that -- but applications fail all the time. It was Microsoft's oversight to put it on Windows critical path. And banks/airlines etc were hit hard because their _Windows_ didn't boot, not because of an application crash on a perfectly working Windows.

hulitu 2y ago
I think they said it was a windows driver, not a normal application. Running crap in kernel mode does not end well on any OS.
- concerned_user 2y ago
  Yes it is a driver which is signed and tested by Microsoft. Driver allows to run arbitrary unsigned code. Why is that allowed?
  - cyberpunk 2y ago
    The driver is some kind of AV/Signature detection hook. E.g check every open() for this list of checksums and refuse to open known viruses style system. The 'update' was a borked definition file which triggered a bug in that system. It's not code execution without signing, and I think probably they do want these files to be updated hands free. The real problem was the lack of testing, rather than the actual mechanism I think.
  - Joker_vD 2y ago
    ...you want Microsoft to forbid you from running certain kinds of programs on your own machine, even if you really, really insist on it, do I understand you correctly?
    - hpen 2y ago
      More like: "...you want Microsoft to forbid you from running certain kinds of programs (with gaping security holes / processes) on your own machine" YES

btbuildem 2y ago

The takeaway from this article seems to be: buy crowdstrike shares, because major corps are unable to make any changes, and will continue to pay licensing fees for this "service" for the foreseeable future.

deleted 2y ago
[deleted]
tootie 2y ago
This is going to crush their sales pipeline and lead to at least a few attempting a migration off. Crowdstrike is unlikely to go out of business, but this is not a good time to buy.
- nkassis 2y ago
  SolarWinds comes to mind they haven't fully recovered but they are still around and kicking.
- alephnerd 2y ago
  Safe Harbor: Don't follow random internet commentators opinions on public markets. This is just an opinion and not advice. I disagree. Long term, the fundamentals of CRWD continue to remain unabated. Endpoint protection is still a critical need no matter what - for every bug like CRWD, there's always a company you can point to who's operations were shut down due to an attack. CRWD skimped on QA and customer support, but long term there aren't many other vendors that can provide a similar service, and CRWD is large enough to pull a PANW and M&A into entirely new segments (eg. DSPM with Flow Security, Observability/Data Lake with Humio, ASPM with Bionic) along with greenfield category makers like Charlotte AI for AI Security and AI EDR. There will be short term pain for CRWD's Windows endpoint business with churn to MDE, SentinelOne, Tanium, etc but they have enough dry powder and a diversified security portfolio that they can safely recover within a year at most. > crush their sales pipeline With CRWD sized companies, most of their revenue comes from multi-year contracts and renewals. They'll probably have a decently large layoff in the sales org, but enterprise sales tends to be fairly stable due to contract sizes along with riders about liability

Retr0id 2y ago

> For historical reasons, that area where almost everything executes is called “userspace.” It's an old term at this point, but I don't think the reasons for it being called "userspace" have changed or become outdated since then, so I wouldn't call them historic per se.

deleted 2y ago
[deleted]
Macha 2y ago
Things have gotten messier with virtualization, containerisation, hypervisors etc. The internet loves to produce pedants to argue the post should go into the finer points of these even when it's not relevant to the message. And so people like the author have a defensive reflex to throw in some language to bounce the pedants away.
SoftTalker 2y ago
I used to like Patrick's posts but lately they are way to long and full of irrelevant minutia. Decide who you're writing for, and write to that audience.
- rescbr 2y ago
  Some of his audience likes the irrelevant minutia.
- rozenmd 2y ago
  > Decide who you're writing for, and write to that audience. He has, and he does.
- arduanika 2y ago
  Congrats, you've been screenshotted and tweeted by him! "In which an HN commenter offers me writing advice but fails to understand the implication of second sentence" https://x.com/patio11/status/1818757982706139297 https://x.com/patio11/status/1818757982706139297
shadowgovt 2y ago

waihtis 2y ago

Regulations are a big reason why this happened, sure, but also it hit the companies with great security budgets more. Hospitals, for instance, weren't that widely affected as they barely have any money to buy security tooling. Silver linings and all that, I guess.

cookiengineer 2y ago
> Hospitals Everybody seems to be quick to forget about WannaCry.
- c0balt 2y ago
  Wannacry was not an accident. It was inarguably an intentional attach against general IT infrastructure instead of a borked update.
- waihtis 2y ago
  That really just proves my point.

toddmorey 2y ago

Was anyone else surprised how little disruption they personally experienced? I had braced for impact that weekend. But all my flights were perfectly on time, all my banking worked, providers worked, and sites & resources were available. I don’t know if I somehow just have little exposure to Windows in my life or if there’s an untold resiliency story for the global internet in the face of such a massive outage. All I can say is THANK YOU to all the unsung heroes who answered the call and worked their butts off. Infrastructure doesn’t work without you. We see you & we thank you!

marcosdumay 2y ago
People found a really quick workaround. It would take a couple more days to fix if there wasn't any.
davio 2y ago
I was unaffected on my work laptop. One of my coworkers is a long-timer and said when the company first got laptops there was a huge "OMG leave your laptops on overnight" push to make sure updates were applied. I always at least sleep if not shut-down after work so I guess I missed out
- doubled112 2y ago
  I know at least one person who "survived" while her coworker's laptops were down. My first question was "do you shut your machine off at the end of the day?" She did, and that's probably why about half of her office was affected, and the other half was not. Can't update it if it isn't on.
blackoil 2y ago
IIRC only 5% of Windows machines were affected. So, it is very probable that most people just saw the news but have no real impact on them. Some had minor and maybe memorable impact, like Indian airlines giving handwritten boarding passes.
- bostik 2y ago

hpen 2y ago

We blame car manufacturers for defects from suppliers, but we don't blame platform manufacturers (Microsoft) for holes in their architecture?

SirMittens 2y ago
I think that's the wrong analogy. A more correct one would be "Should we blame a car company for a broken engine, that was modified after it was sold to you?". A kernel level driver from a 3rd party is something that you willingly add to the OS, it wasn't there. Just because windows allow you to do it, doesn't mean you should. I mean, you can apply some dangerous mods to your car's engine, but you probably shouldn't, and if you do, it's your responsibility, not the car company.
- hpen 2y ago
  Does crowdstrike void the warranty like an engine add on?
  - vel0city 2y ago
    If you had a support contract with Microsoft for your Windows installs and CrowdStrike is breaking your system they'll tell you to go talk to CrowdStrike, yes.
    - hpen 2y ago
      Ok I didn't realize that crowdstrike was more of competitor or maybe a hacky add-on (like a NOS). I was under the impression that it was something more in cooperation (not owned by or anything) but with Microsoft in terms of market support.
      - vel0city 2y ago
        CrowdStrike absolutely is a competitor to Microsoft. Microsoft sells licensed software in the exact same market as CrowdStrike. Microsoft even sells Microsoft Defender for MacOS and Linux. They're direct competitors. https://www.microsoft.com/en-us/security/business/endpoint-security/microsoft-defender-endpoint?msockid=0fe5b142a836612518bda1dba9976023 https://www.microsoft.com/en-us/security/business/endpoint-s... https://learn.microsoft.com/en-us/defender-endpoint/non-windows https://learn.microsoft.com/en-us/defender-endpoint/non-wind...

deleted 2y ago

[deleted]

jmuguy 2y ago

Maybe the IT departments at the affected orgs take solace in the fact that so many other orgs had issues that the heat is off - but in my opinion this was still a failure of IT itself. There's no reason that update should have been pushed automatically to the entire fleet. If Crowdstrike's software doesn't give you a way to rollout updates on a portion of your network before the entire fleet, it shouldn't be used.

candiddevmike 2y ago
The update bypassed the controls orgs had in place to defer/schedule updates, AFAIK.
- jmuguy 2y ago
  I've had trouble nailing down if thats the case from searching around online. And if thats true - thats absolutely on Crowdstrike. And that behavior should disqualify it from being used on critical systems. I imagine this incident will cause a lot of teams to consider just what can happen automatically on their systems.
  - tantalor 2y ago
    [flagged]
    - lucianbr 2y ago
      How did 911 services go down then? Whatever system caused that, should be by definition critical, imho.
    - Sohcahtoa82 2y ago
      You're living in a different reality. I can't fathom how anybody could legitimately make that claim. Even if you're defining "critical system" as "critical to humans" and not "critical to the business", then sure, you can say "Airlines aren't critical" and for most passengers, yeah, you're probably right. Most industries aren't critical, so businesses being ground to a halt doesn't matter for the consumers. But 911 systems were affected, and those are certainly critical to humans. If 911 doesn't work, ambulances and fire trucks can't be dispatched, and people die. EDIT: Computers attached to hospital beds, including trauma surgery rooms, were affected. I'm really curious what you think defines a critical system.

__MatrixMan__ 2y ago

I like the technical stuff here. I'm not so sure about this: > money is core societal infrastructure, like the power grid and transportation systems are. It would be really bad if hackers working for a foreign government could just turn off money. Sure, it would be inconvenient in the short term. But I think the current design is holding us back. I suspect that most of us would have more to gain than to lose if we managed to shut off money-as-we-know-it and keep it off for long enough to iterate on alternatives. Any design that even tried to step beyond "well that's how we've always done it" would likely land somewhere better than what we're doing. Much has changed since Alexander Hamilton.

gadders 2y ago
In the short term people would probably starve to death.
- Joker_vD 2y ago
  Probably not. A competent government could install temporary rationing for the most essential goods such as food. It happened through the the whole of the 1917—1920 Russian revolution, with four or five kinds of paper money being circulated around, and the urban population managed through it only if barely. That government was much less competent than the US government is today.
  - mminer237 2y ago
    I mean, millions still starved during the revolution, even with the American Relief Administration feeding 10% of the country.
    - Joker_vD 2y ago
      In the rural areas, mind you. That's one of the most appalling thing about famines in the XIX-XX, that they hit the countryside heavier than they hit the cities.
  - deleted 2y ago

kristaps 2y ago

The article specifically mentions US banks and as I personally didn't see any disruption over here - is there (anec)data on how popular CrowdStrike is in the US vs the EU?

Muromec 2y ago
Can't have disruption from CrowdStrike if you run on IBM mainframes with cobol coz your math only opens gates for new technologies once in 25 years.
- MattSayar 2y ago
  Oh wow an Anathem reference! To answer the question, CrowdStrike is a global company with thousands of employees around the world. Not sure why the EU wasn't hit as hard.
Ekaros 2y ago
Might be question what type of disruption it is. Transfers and web bank is likely to work. Branches offices and ATMs might have issues. So if you try to do anything in person or negotiate anything with workers in bank there could be issues.

adrr 2y ago

Did it really hit banks hard? Core banking systems don't run windows, they run on mainframes typically on IBM z/OS. I know it hit the financial firms hard and knocked out their trading systems but I don't know of any major bank losing their core bank system due to crowdstrike. Australia got hit hard because they modernized their bank systems and now most are cloud based. I am not aware of any major bank running their core systems on the cloud or on windows.

tempodox 2y ago
> they modernized their bank systems You mean they made them more vulnerable?
josephthejoe 2y ago
[dead]

taeric 2y ago

Any explanation that doesn't boil this down to "software required by corporate policy checklist not written by technical team" is almost certainly missing something here. This is almost definitionally policy capture by a security team and the all too common consequences that attach. The section that goes over why this wasn't federally pushed is largely accurate, mind. Not all capture is at the federal level. Is why you can get frustrated with customer support for asking you a checklist of unrelated questions to the problem you have called in. And the super frustrating thing is that these checklists are often very effective for why they exist.

voytec 2y ago

> Another way is if it has recently joined a botnet orchestrated from a geopolitical adversary of the United States after one of your junior programmers decided to install warez because the six figure annual salary was too little to fund their video game habit. Fictional statements like this make me reluctant to read further, and ignore source of such "news" in the future.

davidgerard 2y ago
what makes you think it was fictional? also, bragging about your inability to read text seems an odd way to interact.
- voytec 2y ago
  Bragging? Reluctant==unable?
bdamm 2y ago
It's obviously fictional, but let's call it contemporary drama based on a true story. I thought the point was well made. The author already noted this was a handwaving segment.
samspot 2y ago
I got in trouble for something like this early in my career (running bittorrent over my work vpn).

bob1029 2y ago

I feel like this only impacted the larger banks. I've heard absolutely no explosion noises coming from smaller institutions. The effect of regulations and their enforcement is felt differently across the spectrum. There is something to be said for a diverse banking industry when it comes to this kind of problem. Also, this event is a powerful argument for keeping the core systems on unusual mainframe architectures. I think building a bank core on windows would be a really bad choice, but some vendors have already done this.

MadVikingGod 2y ago

While reading this I was struck with an interesting question: What risk does any particular software vendor pose to an industry at large? For example (making up numbers here): if 75% of all airline computers have croudstrike falcon installed that seems like a very concentrated risk. I actually wouldn't be surprised if we had this we would see really high concentrations of a small number of vendors in any industry.

anticristi 2y ago
The EU DORA regulation (Digital Operational Resilience Act for Financial Entities) has explicit provisions to avoid concentration risks. I heard a story that a bank was forced to use Google Cloud, because two other banks were already on AWS and Azure.
saltminer 2y ago
Alternatively, if Oracle hikes the price on an industry-specific product by 75%, how much of that industry goes under?

shadowgovt 2y ago

Just as a general comment on this whole affair: This would be the third incident I'm familiar with of a file of entirely zeroes breaking something big. Folks, as much as we wish it weren't true, null comes up all the damn time, and if you don't have tests trying to force-feed null into your system in novel and exciting ways, production will demonstrate them for you. Never assume 'zero' (for whatever form zero takes in context) can't be an input.

tempodox 2y ago
As long as the botchers get away with impunity, they won't “waste” resources on higher standards.

saltminer 2y ago

> This created a minor emergency for me, because it was an other-than-minor emergency for some contractors I was working with. > Many contractors are small businesses. Many small businesses are very thinly capitalized. Many employees of small businesses are extremely dependent on receiving compensation exactly on payday and not after it. And so, while many people in Chicago were basically unaffected on that Friday because their money kept working (on mobile apps, via Venmo/Cash App, via credit cards, etc), cash-dependent people got an enormous wrench thrown into their plans. I never really thought about not having to worry about cashflow problems as a privilege before, but it makes sense, considering having access to the banking system to begin with is a privilege. I remember my bank's website and app were offline, but card processing was unaffected - you could still swipe your cards at retailers. For me, the disruption was a minor annoyance since I couldn't check my balance, but I imagine many people were probably panicking about making rent and buying groceries while everything was playing out.

HeyLaughingBoy 2y ago
The really admirable thing about this is that Patrick acknowledged that it was "an other-than-minor emergency" for the contractors and took steps to ensure that they were paid rapidly. In a similar situation many people would have shrugged and taken an attitude of "sorry, bank's down. I'll pay you when it comes back up."

HeyLaughingBoy 2y ago

> Configuration bugs are a disturbingly large portion of engineering decisions which cause outages I work in medical device software -- the stuff that runs on machines in hospital labs, ER's or at patient bedside. The first "ohmigod do we need to recall this?" bug I remember was an innocuous piece of code that was inserted to debug a specific problem, but which was supposed to be disabled in the "non-debug" configuration. Then somehow, the software update shipped with a change to the configuration file that enabled that code to run. Timing-critical debug code running on a real-time system with a hard deadline is a recipe for disaster. Thankfully, we got out of that pretty easily before it affected more than a small handful of users, but things could have been a lot worse.