4 ms·

Why are so many mission critical hardware connected systems connected to the internet at all or getting automatic updates? This is just basic IT common sense.

by _yb2s 2y ago

Why are so many mission critical hardware connected systems connected to the internet at all or getting automatic updates? This is just basic IT common sense. You only do updates during a planned outage, after doing an easily reversible backup, or you have two redundant systems in rotation and update and test the spare first. Critical systems connected to things like medical equipment should have no internet connectivity, and need no security updates. I follow all of this in my own home so a bad update doesn’t ruin my work day… how do big companies with professional IT not know this stuff?

gedy 2y ago
> Why are so many mission critical hardware connected systems connected to the internet at all or getting automatic updates? Because it lets them "scale" by having fewer and cheaper offsite IT and contractors to manage vs hiring pesky onsite employees.
basch 2y ago
You do that for antivirus definition updates?
- mckn1ght 2y ago
  Probably, implicitly. Have automated regular backups, and don’t let your AV automatically update, or even if it does, don’t log into all your computers simultaneously. If you update/login serially, then the first BSOD would maybe prevent you from doing the same thing on the other (or possibly, send you running to the other to accomplish your task, and BSODing that one too!) But yeah this is one reason why I don’t have automatic updates enabled for anything, the other major one being that companies just can’t resist screwing with their UIs.
- _yb2s 2y ago
  I’m not an IT professional, but I don’t use antivirus software on my personal macs and linux machines- I do regular rotated physical backups, and only install software digitally signed by trusted sources and well reviewed Pirate Bay accounts (that's a joke :-). My only windows machine is what I would classify as a mission critical hardware connected/control device, an old Windows 8 tablet I use for car diagnostics- I do not connect it to the internet, and never perform updates on it. I am an academic and use a lot of old multi-million dollar scientific instruments which have old versions of windows controlling them. They work forever if you don't network them, but the first time you do, someone opens up a browser to check their social media, and the entire system will fail quickly.
- barryrandall 2y ago
  Yes. In an environment where you have so many clients that they can DDoS the antivirus management server, you have to stagger the update schedule anyway. The way we set it up, sysadmins/help desk/dev deployments updated on day 1, all IT workstations/test deployments updated on day 2, and all workstations/staging/production deployments on day 3.
  - genewitch 2y ago
    what happens if there's a 0-day RCE? 72 hours of your production systems hanging out in the open...
    - barryrandall 2y ago
      The schedules are shockingly easy to adjust.
vel0city 2y ago
CrowdStrike let's you create update strategies and rollout groups. This update bypassed all of those settings.
- juio 2y ago
  [flagged]
  - Brechreiz 2y ago
    This worked for me!
- _yb2s 2y ago
  Well that context makes it make a little more sense... I still wouldn't be trusting a service like that for mission critical hardware that shouldn't be connected to the internet in the first place. The question with these types of services is: is your goal to keep the system as reliable as possible, or to be able to place the blame on a 3rd party when it goes down? If it's a critical safety system that human lives depend on, the answer better be the former.
  - basch 2y ago
    How do you keep an airline ticketing system offline? How would anybody book tickets without access to the databases?
    - zamubafoo 2y ago
      Whitelist the persistent store?
    - nsguy 2y ago
      you don't need to airgap it. just limit the access to the specific APIs/access to the database and block everything else. CrowdStrike won't be able to upgrade itself through your database API...
  - MetaWhirledPeas 2y ago
asynchronous 2y ago
What people aren’t understanding is MOST of the outage isn’t caused by a crowdstrike install itself, it’s caused because something upstream of it (a critical application server) is what got borked, and that’s having a domino effect on everything else.

by _yb2s 2y ago

gedy 2y ago

> Why are so many mission critical hardware connected systems connected to the internet at all or getting automatic updates? Because it lets them "scale" by having fewer and cheaper offsite IT and contractors to manage vs hiring pesky onsite employees.

basch 2y ago

You do that for antivirus definition updates?

mckn1ght 2y ago
Probably, implicitly. Have automated regular backups, and don’t let your AV automatically update, or even if it does, don’t log into all your computers simultaneously. If you update/login serially, then the first BSOD would maybe prevent you from doing the same thing on the other (or possibly, send you running to the other to accomplish your task, and BSODing that one too!) But yeah this is one reason why I don’t have automatic updates enabled for anything, the other major one being that companies just can’t resist screwing with their UIs.
_yb2s 2y ago
I’m not an IT professional, but I don’t use antivirus software on my personal macs and linux machines- I do regular rotated physical backups, and only install software digitally signed by trusted sources and well reviewed Pirate Bay accounts (that's a joke :-). My only windows machine is what I would classify as a mission critical hardware connected/control device, an old Windows 8 tablet I use for car diagnostics- I do not connect it to the internet, and never perform updates on it. I am an academic and use a lot of old multi-million dollar scientific instruments which have old versions of windows controlling them. They work forever if you don't network them, but the first time you do, someone opens up a browser to check their social media, and the entire system will fail quickly.
barryrandall 2y ago
Yes. In an environment where you have so many clients that they can DDoS the antivirus management server, you have to stagger the update schedule anyway. The way we set it up, sysadmins/help desk/dev deployments updated on day 1, all IT workstations/test deployments updated on day 2, and all workstations/staging/production deployments on day 3.
- genewitch 2y ago
  what happens if there's a 0-day RCE? 72 hours of your production systems hanging out in the open...
  - barryrandall 2y ago
    The schedules are shockingly easy to adjust.

vel0city 2y ago

CrowdStrike let's you create update strategies and rollout groups. This update bypassed all of those settings.

juio 2y ago
[flagged]
- Brechreiz 2y ago
  This worked for me!
_yb2s 2y ago
Well that context makes it make a little more sense... I still wouldn't be trusting a service like that for mission critical hardware that shouldn't be connected to the internet in the first place. The question with these types of services is: is your goal to keep the system as reliable as possible, or to be able to place the blame on a 3rd party when it goes down? If it's a critical safety system that human lives depend on, the answer better be the former.
- basch 2y ago
  How do you keep an airline ticketing system offline? How would anybody book tickets without access to the databases?
  - zamubafoo 2y ago
    Whitelist the persistent store?
  - nsguy 2y ago
    you don't need to airgap it. just limit the access to the specific APIs/access to the database and block everything else. CrowdStrike won't be able to upgrade itself through your database API...
- MetaWhirledPeas 2y ago

asynchronous 2y ago

What people aren’t understanding is MOST of the outage isn’t caused by a crowdstrike install itself, it’s caused because something upstream of it (a critical application server) is what got borked, and that’s having a domino effect on everything else.