7 ms·

"Took down our entire emergency department as we were treating a heart attack. 911 down for our state too." Why would Windows systems be anywhere near critical

by rsync 2y ago

"Took down our entire emergency department as we were treating a heart attack. 911 down for our state too." Why would Windows systems be anywhere near critical infra ? Heart attacks and 911 are not things you build with Windows based systems. We understood this 25 years ago.

Wytwwww 2y ago
Well you can use stupid broken software with any OS, not just Windows. Isn't CrowdStrike Falcon available on Linux, is there any reason why couldn't they have introduced a similar bug and similar consequences there?
- tw04 2y ago
  None. There are a bunch of folks here who clearly haven’t spent a day in enterprise IT proclaiming Linux would’ve saved the day. 30 seconds of research would’ve lead them to discover crowdstrike also runs on Linux and has created similar problems on Linux in the past.
  - rowanG077 2y ago
    Oh could you link me the source of the claim that all linux clients of crowdstrike went down all at once? I'm very interested to hear it.
  - p_l 2y ago
    It's even better when you get told about the magical superiority of apple for that... ... Except Apple pretty much pushes you to run such tools just to get reasonable management key alone things like real-time integrity monitoring of important files (Crowdstrike in $DAYJOB[-1] is how security knew to ask whether it was me or something else that edited PAM config for sudo on corporate Mac)
    - nikau 2y ago
      Enterprise mac always follows the same pattern, users proclaim its superiority while its off the radar, then it gets mcaffee, carbon black, airlock, and a bunch of other garbage tooling installed and runs as poorly as enterprise Windows. The best corporate dev platform at moment is WSL2 - most of the activity inside the WSL2 vm isn't monitored by the windows tooling so performance is fast. Eventually security will start to mandate agents inside the WSL2 instance, but at the moment most orgs dont.
  - olddustytrail 2y ago
    No it couldn't. Crowdstrike on Linux uses eBPF and therefore can't cause a kernel panic (which is the fundamental issue here).
    - tapoxi 2y ago
      Happened last month: https://access.redhat.com/solutions/7068083 https://access.redhat.com/solutions/7068083
Nathanael_M 2y ago
Because Windows is accessible and Linux requires uncommon expertise and short term cost that is just not practical for lots of places. Goodluck teaching administrators an entirely new ecosystem, goodluck finding software off the shelf for Linux. Bespoke is expensive, expertise is rare, Linux is sadly niche.
- briandear 2y ago
  No. The problem isn’t expertise — it’s CIOs that started their career in the 1990s and haven’t kept up with the times. I had to explain why we wanted PostgreSQL instead of MS SQL server. I shouldn’t have to have that conversation with an executive that should theoretically be a highly experienced expert. We also have CIOs that have MBAs but not actual background in software. (I happen to have an MBA but I also have 15+ years of development experience.) My point is CIOs generally know “business” and they know how to listen to pitches from “Enterprise” software companies — but they don’t actually have real-world experience using the stuff they’re forcing upon the org. I recently did a project with a company that wanted to move their app to Azure from AWS — not for any good technical reason but just because “we already use Microsoft everywhere else.” Completely stupid. S3 and Azure Blob don’t work the same way. MCS and AWS SES also don’t work the same way — but we made the switch not even for reasons of money, but because some Microsoft salesman convinced the CIO that their solution was better. Similar to why many Jira orgs force Bitbucket on developers — they listen to vendors rather than the people that have to use this stuff.
  - Nathanael_M 2y ago
    That’s so infuriating. But, while the people in your story sound dumb, they still sound way more technically literate than 95% of society. Azure is blue, AWS is followed by OME. Teach a 60 year old industrial powertrain salesman to use Linux and to redevelop their 20 year old business software for a different platform. Also explain why it’s worth spending food, house, and truck money on it. Finally, local IT companies are often incompetent. You get entire towns worth of government and business managed by a handful of complacent, incompetent local IT companies. This is a ridiculously common scenario. It totally sucks, and it’s just how it is.
sentientslug 2y ago
It seems like you’ve never worked with critical infra. Most of it runs on 6 to 10 year old unpatched versions of Windows…
- rsync 2y ago
  "It seems like you’ve never worked with critical infra." My entire career has been spent building, and maintaining, critical infra.[1] Further, in my volunteer time, I come into contact with medical, dispatch and life-safety systems and equipment built on Windows and my question remains the same: Why is Windows anywhere near critical infra ? Just because it is common doesn't mean it's any less shameful and inadequate. I repeat: We've fully understood these risks and frailties for 25 years. [1] As a craft, and a passion - not because of "exciting career opportunities in IT".
  - sgarland 2y ago
    Is this the rsync.net HN account? If so, lmao @ the comment you replied to. > As a craft, and a passion I believe you’ve nailed the core problem. Many people in tech are not in it because they genuinely love it, do it in their off time, and so on. Companies, doubly so. I get it, you have to make money, but IME, there is a WORLD of difference in ability and self-solving ability between those who love this shit, and those who just do it for the money. What’s worse is that actual fundamental knowledge is being lost. I’ve tried at multiple companies to shift DBs off of RDS / Aurora and onto at the very least, EC2s. “We don’t have the personnel to support that.” “Me. I do this at home, for fun. I have a rack. I run ZFS. Literally everything in this RFC, I know how to do.” “Well, we don’t have anyone else.” And that’s the damn tragedy. I can count on one hand the number of people I know with a homelab who are doing anything other than storing media. But you try telling people that they should know how to administer Linux before they know how to administer a K8s cluster, and they look at you like you’re an idiot.
    - nikau 2y ago
      Also a lot of the passionate security people such as myself moved on to other fields as it has just become bullshit artists sucking on the vendors teat and filling out risk matrix sheets, but no accountability when their risk assessments invariably turn out to be wrong.
charles_f 2y ago
> Why would Windows systems be anywhere near critical infra ? Why would computers be anywhere near critical infra? This sounds like something that should failsafe, the control system goes down but the thing keeps running. If power goes down, hospitals have generator backups, it seems weird that computers would not be in the same situation
orbillius 2y ago
> Why would Windows systems be anywhere near critical infra ? This is just a guess, but maybe the client machines are windows. So maybe there are servers connected to phone lines or medical equipment, but the doctors and EMS are looking at the data on windows machines.
freehorse 2y ago
I do not think windows is the problem here. The problem is that equipment that is critical infrastructure being connected to the internet, imo. There is little reason for a lot of computers in some settings to be connected to the internet, except for convenience or negligence. If data transfer needs to be done, it can happen through another computer. Some systems should exist on a (more or less) isolated network at best. Too often we do not really understand the risk of a device being connected to the internet, until something like this happens.
- et2o 2y ago
  You have no idea how a hospital or modern medicine works. It needs to be online.
  - freehorse 2y ago
    Why would a machine that is required for a MRI machine to work (as one of the examples given in the thread here) need to be online? I understand about logging, though even then I think it is too risky. Do all these machines _really_ need to be online, or just nobody bothered after all the times something happened or, even worse, software companies profit in certain ways and would not want to change their models? Can we imagine no other way to do things apart from connecting everything to some server wherever that is?
    - compiler-guy 2y ago
      You don’t print the images an MRI produced, you transmit them to the people who can interpret them, and they are almost never in the same room as the big machine, and sometimes they need to be called up in a different office altogether.
      - freehorse 2y ago
        The comment [0] mentioned that they could not get at all the mri outputs even with the radiologist coming on site. Obviously, software that was processing/exporting the data was running on a computer that was connected online, if not requiring internet connection itself. Data transfer can happen from another computer than the one the data is processed/obtained. Less convenient, but this is common practice in many other places for security and other reasons. [0] https://news.ycombinator.com/item?id=41009018 https://news.ycombinator.com/item?id=41009018
nO0b 2y ago
> Why would Windows systems be anywhere near critical infra ? maybe Heartbleed or the xzUtils debacles convinced them to switch.

by rsync 2y ago

Wytwwww 2y ago

Well you can use stupid broken software with any OS, not just Windows. Isn't CrowdStrike Falcon available on Linux, is there any reason why couldn't they have introduced a similar bug and similar consequences there?

tw04 2y ago
None. There are a bunch of folks here who clearly haven’t spent a day in enterprise IT proclaiming Linux would’ve saved the day. 30 seconds of research would’ve lead them to discover crowdstrike also runs on Linux and has created similar problems on Linux in the past.
- rowanG077 2y ago
  Oh could you link me the source of the claim that all linux clients of crowdstrike went down all at once? I'm very interested to hear it.
- p_l 2y ago
  It's even better when you get told about the magical superiority of apple for that... ... Except Apple pretty much pushes you to run such tools just to get reasonable management key alone things like real-time integrity monitoring of important files (Crowdstrike in $DAYJOB[-1] is how security knew to ask whether it was me or something else that edited PAM config for sudo on corporate Mac)
  - nikau 2y ago
    Enterprise mac always follows the same pattern, users proclaim its superiority while its off the radar, then it gets mcaffee, carbon black, airlock, and a bunch of other garbage tooling installed and runs as poorly as enterprise Windows. The best corporate dev platform at moment is WSL2 - most of the activity inside the WSL2 vm isn't monitored by the windows tooling so performance is fast. Eventually security will start to mandate agents inside the WSL2 instance, but at the moment most orgs dont.
- olddustytrail 2y ago
  No it couldn't. Crowdstrike on Linux uses eBPF and therefore can't cause a kernel panic (which is the fundamental issue here).
  - tapoxi 2y ago
    Happened last month: https://access.redhat.com/solutions/7068083 https://access.redhat.com/solutions/7068083

Nathanael_M 2y ago

Because Windows is accessible and Linux requires uncommon expertise and short term cost that is just not practical for lots of places. Goodluck teaching administrators an entirely new ecosystem, goodluck finding software off the shelf for Linux. Bespoke is expensive, expertise is rare, Linux is sadly niche.

briandear 2y ago
No. The problem isn’t expertise — it’s CIOs that started their career in the 1990s and haven’t kept up with the times. I had to explain why we wanted PostgreSQL instead of MS SQL server. I shouldn’t have to have that conversation with an executive that should theoretically be a highly experienced expert. We also have CIOs that have MBAs but not actual background in software. (I happen to have an MBA but I also have 15+ years of development experience.) My point is CIOs generally know “business” and they know how to listen to pitches from “Enterprise” software companies — but they don’t actually have real-world experience using the stuff they’re forcing upon the org. I recently did a project with a company that wanted to move their app to Azure from AWS — not for any good technical reason but just because “we already use Microsoft everywhere else.” Completely stupid. S3 and Azure Blob don’t work the same way. MCS and AWS SES also don’t work the same way — but we made the switch not even for reasons of money, but because some Microsoft salesman convinced the CIO that their solution was better. Similar to why many Jira orgs force Bitbucket on developers — they listen to vendors rather than the people that have to use this stuff.
- Nathanael_M 2y ago
  That’s so infuriating. But, while the people in your story sound dumb, they still sound way more technically literate than 95% of society. Azure is blue, AWS is followed by OME. Teach a 60 year old industrial powertrain salesman to use Linux and to redevelop their 20 year old business software for a different platform. Also explain why it’s worth spending food, house, and truck money on it. Finally, local IT companies are often incompetent. You get entire towns worth of government and business managed by a handful of complacent, incompetent local IT companies. This is a ridiculously common scenario. It totally sucks, and it’s just how it is.

sentientslug 2y ago

It seems like you’ve never worked with critical infra. Most of it runs on 6 to 10 year old unpatched versions of Windows…

rsync 2y ago
"It seems like you’ve never worked with critical infra." My entire career has been spent building, and maintaining, critical infra.[1] Further, in my volunteer time, I come into contact with medical, dispatch and life-safety systems and equipment built on Windows and my question remains the same: Why is Windows anywhere near critical infra ? Just because it is common doesn't mean it's any less shameful and inadequate. I repeat: We've fully understood these risks and frailties for 25 years. [1] As a craft, and a passion - not because of "exciting career opportunities in IT".
- sgarland 2y ago
  Is this the rsync.net HN account? If so, lmao @ the comment you replied to. > As a craft, and a passion I believe you’ve nailed the core problem. Many people in tech are not in it because they genuinely love it, do it in their off time, and so on. Companies, doubly so. I get it, you have to make money, but IME, there is a WORLD of difference in ability and self-solving ability between those who love this shit, and those who just do it for the money. What’s worse is that actual fundamental knowledge is being lost. I’ve tried at multiple companies to shift DBs off of RDS / Aurora and onto at the very least, EC2s. “We don’t have the personnel to support that.” “Me. I do this at home, for fun. I have a rack. I run ZFS. Literally everything in this RFC, I know how to do.” “Well, we don’t have anyone else.” And that’s the damn tragedy. I can count on one hand the number of people I know with a homelab who are doing anything other than storing media. But you try telling people that they should know how to administer Linux before they know how to administer a K8s cluster, and they look at you like you’re an idiot.
  - nikau 2y ago
    Also a lot of the passionate security people such as myself moved on to other fields as it has just become bullshit artists sucking on the vendors teat and filling out risk matrix sheets, but no accountability when their risk assessments invariably turn out to be wrong.

charles_f 2y ago

> Why would Windows systems be anywhere near critical infra ? Why would computers be anywhere near critical infra? This sounds like something that should failsafe, the control system goes down but the thing keeps running. If power goes down, hospitals have generator backups, it seems weird that computers would not be in the same situation

orbillius 2y ago

> Why would Windows systems be anywhere near critical infra ? This is just a guess, but maybe the client machines are windows. So maybe there are servers connected to phone lines or medical equipment, but the doctors and EMS are looking at the data on windows machines.

freehorse 2y ago

I do not think windows is the problem here. The problem is that equipment that is critical infrastructure being connected to the internet, imo. There is little reason for a lot of computers in some settings to be connected to the internet, except for convenience or negligence. If data transfer needs to be done, it can happen through another computer. Some systems should exist on a (more or less) isolated network at best. Too often we do not really understand the risk of a device being connected to the internet, until something like this happens.

et2o 2y ago
You have no idea how a hospital or modern medicine works. It needs to be online.
- freehorse 2y ago
  Why would a machine that is required for a MRI machine to work (as one of the examples given in the thread here) need to be online? I understand about logging, though even then I think it is too risky. Do all these machines _really_ need to be online, or just nobody bothered after all the times something happened or, even worse, software companies profit in certain ways and would not want to change their models? Can we imagine no other way to do things apart from connecting everything to some server wherever that is?
  - compiler-guy 2y ago
    You don’t print the images an MRI produced, you transmit them to the people who can interpret them, and they are almost never in the same room as the big machine, and sometimes they need to be called up in a different office altogether.
    - freehorse 2y ago
      The comment [0] mentioned that they could not get at all the mri outputs even with the radiologist coming on site. Obviously, software that was processing/exporting the data was running on a computer that was connected online, if not requiring internet connection itself. Data transfer can happen from another computer than the one the data is processed/obtained. Less convenient, but this is common practice in many other places for security and other reasons. [0] https://news.ycombinator.com/item?id=41009018 https://news.ycombinator.com/item?id=41009018

nO0b 2y ago

> Why would Windows systems be anywhere near critical infra ? maybe Heartbleed or the xzUtils debacles convinced them to switch.