10 ms·

"Internet-accessble industrial control systems have been a problem for many years now ..." They are a problem the way drunk driving is a problem. You just don

by rsync 6y ago

"Internet-accessble industrial control systems have been a problem for many years now ..." They are a problem the way drunk driving is a problem. You just don't ever do it. Ever. No cyber security products are needed. No budget required. These "startups in the ICS space" are like turbotax/HRBlock: only continued idiocy allows their business model to exist.

deleted 6y ago
[deleted]
karmicthreat 6y ago
I think these days it is becoming a business need though. These systems are made by vendors who probably need remote access. Also if the plant has a relatively unsophisticated IT department then someone is just going to shove an AP in the ceiling so they can check things when they get called at 1AM. Several ICS vendors like Tosibox and EWON make devices to accomplish this. I think Tosi has the more secure model, though I hate their proprietary dongles. VPNs are also used pretty successfully here. Several large companies also don't let you directly connect to anything. You vpn in and connect to a machine with Citrix and then you can use whatever was setup for your there. Usually whatever version of Logix/Studio 5000 the plant is on. You have to talk to someone in IT to get your files moved in/out. I think Amazon went a different direction and uses Versiondog to monitor their automation systems and check for changes. I don't work there or know anyone on their automation team so I'm not aware of the details. Still, I think you can have external access and be secure. You just need to balance things out with your business needs.
- EvanAnderson 6y ago
  VPNs are also used pretty successfully here. Several large companies also don't let you directly connect to anything. You vpn in and connect to a machine with Citrix and then you can use whatever was setup for your there. Usually whatever version of Logix/Studio 5000 the plant is on. I support an environment almost exactly like this (albeit in a small manufacturing company). I don't love having one of the controls networks attached, in any way, to the LAN, but I understand the business requirements justify it. It happens that there's a controls system running devices that could cause massive environmental impact in some malfunction scenarios. I am happy to report the plant, being held to account for things like public evacuation plans and hazmat filings with local first responders, has never asked about connecting that network to anything. That would be a walk-out-the-door type scenario for me. I worry that they'd just find somebody who wouldn't have those scruples, though.
  - just_steve_h 6y ago
    > I worry that they'd just find somebody who wouldn't have those scruples, though. If I may speak slightly out of turn to a stranger, using a possible & currently imagined future person less scrupulous does not modify in any way your obligation, however you perceive it, to act ethically.
    - EvanAnderson 6y ago
      I've warned them about my concerns. If I stop working with them, and I have no further knowledge of thir situation, I don't see what else there would be for me to do.
    - Dylan16807 6y ago
      Ohhhhkay? Nobody was suggesting that it does. The person you're replying to specifically said the opposite.
  - karmicthreat 6y ago
    That's an understandable scenario and good on you guys for balancing the risks.
  - uzakov 6y ago
    What are your thoughts on something like this in your line of work? https://cloud.google.com/beyondcorp https://cloud.google.com/beyondcorp
    - EvanAnderson 6y ago
      I may be too much of a simple "IT guy" to grok the deep meaning of BeyondCorp. I read thru some of the various papers when they came out and always came back to the thought "Yeah, that's nice if you have the resources to exert control over that much of your technology stack." I don't have those resources, nor do my Customers. I've got the various mix of Windows, Linux, and embedded devices that the Customer has purchased to serve their business applications. They (and I) don't have the clout or purchasing power to demand application vendors bend to our desires, so I'm left with making the best out of sub-optimal architecture, protocols, etc. Google says, in the BeyondCorp III paper under the heading "Third-Party Software"[1]: Third-party software has frequently proved troublesome, as sometimes it can’t present TLS certificates, and sometimes it assumes direct connectivity. In order to support these tools, we developed a solution to automatically establish encrypted point-to-point tunnels (using a TUN device). The software is unaware of the tunnel, and behaves as if it’s directly connected to the server. So, they just do what I do and throw a VPN at it, albeit a client-to-server VPN serving an individual application rather than a client-to-network VPN like I might. I do my best to segment the networks at my Customer sites, to use default-deny policies between security zones, to authenticate traffic flows to users and devices where possible, and when unable (because of limitations of client software/devices, usually) restrict access by source address. Within each security zone I try to make a worst-case assumption of an attacker getting complete access to the zone (compromising a host within the zone and getting arbitrary network access, for example) with things like private VLANs and host-based firewalls. I have to declare "bankruptcy" in some security zones (usually where there are embedded devices) where I have to rely only on network segmentation because the devices (or vendors) are too "stupid" to have host-based firewall functionality, authentication, encryption, etc. (These are the devices that fall over and die when they get port-scanned, yet somehow end up in mission-critical roles.) I think the harsh reality is that, operating at the scale of small to mid-sized companies, IT and infosec are forced into a lot of bad places by vendors who don't care, and management who are focused on the bottom-line and who don't see security as anything other than something to purchase insurance for. To put it another way: I have to make all this crap work. If I make it too difficult for the end users to work or for the vendors to support I'll be kicked to the curb and they'll find somebody else who will be less "difficult". [1] https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45728.pdf https://storage.googleapis.com/pub-tools-public-publication-...
- closeparen 6y ago
  Remote Desktop was exactly the mechanism here, the attacker used TeamViewer to work the UI on a plant operator’s desktop and he happened to be watching.
  - rootsudo 6y ago
    Seeing reports now that it was indeed Teamviewer! They also have a satellite office over in Clearwater, Florida (which is trying to be like a little bay area copy, v2/3) Interesting, but, Teamviewer also has been exploited and leaked creds, but took three years to confirm it: https://www.bleepingcomputer.com/news/security/teamviewer-confirms-undisclosed-breach-from-2016/ https://www.bleepingcomputer.com/news/security/teamviewer-co... Or, if the client computer browsed a site, it'd actually start open an SMB share on the perps computer: https://www.bleepingcomputer.com/news/security/teamviewer-fixes-bug-that-lets-attackers-access-your-pc/ https://www.bleepingcomputer.com/news/security/teamviewer-fi... and a few other interesting vulnerabilities, hmm.
  - karmicthreat 6y ago
    Citrix is a little more than teamviewer. And can be encapsulated in a VPN as well. Teamviewer on a desktop, probably with a shared credential isn't very secure. Knowing this though, I doubt that it was a teamviewer exploit. My guess would be a disgruntled employee since they knew what to get into to change chemical set points.
    - closeparen 6y ago
      They both expose Remote Desktop to the internet given the proper credentials, and I’m guessing here but I think it’s pretty likely that the attacker had credentials. Whether it was a disgruntled insider, a dumb password, or (most likely) a reused password from a leak somewhere. I would be interested to know if the TeamViewer account in question had 2FA... probably not.
- Veserv 6y ago
  It is not a business "need". These systems have functioned without remote access perfectly well for decades. It is a business "want" and thus must be balanced against any new risks relative to historical risks. The risk of adding remote access to critical systems is the introduction of globally accessible single-point-of-failures. Given the nature of software, such an attack has an unlimited amount of time to be perfected before deployment and when finished can be deployed at effectively zero cost and complete in effectively zero time which provides no meaningful way to respond except with already deployed automated systems. So, the risk added with remote access is the risk of malicious catastrophic total system failure. In this case, the water treatment facility treated the water for ~15,000 residents. In a similar case many years ago [1], a similar event occurred to a water treatment facility that treated the water for ~12,000 residents which resulted in 100 affected individuals before the effects were detected. So, we can reasonably assume that undetected water treatment tampering on a facility serving ~10,000 individuals will result in about ~100 affected individuals before the effects are detected. If there exists a way to tamper with a water treatment facility that would result in deaths for the affected individuals, which is quite likely, then that means the risk of remote access to the water treatment facility is ~100 deaths. So, as a society, we should ask the question: What is the standard of care that should be applied to a system where failure may result in the deaths of 100 people? And any business that wishes to add remote access to such a system must demonstrate to the satisfaction of society that they are taking that degree of care. It is not the role of society or the people to suffer for the convenience of business. And in this case, I am certain that they are not taking an appropriate amount of care. The fact that you honestly suggested that an IT department would shove an AP in the ceiling for their convenience shows just how low our expectations are. In any other industry, such an act would be, in no uncertain terms, criminal negligence. That our standard assumption about the standard of care taken is criminal negligence shows just how far any of these companies is from actually deploying systems that have external access and have adequate security. [1] https://www.spencerma.gov/sites/g/files/vyhlif1246/f/uploads/wateremergency.pdf https://www.spencerma.gov/sites/g/files/vyhlif1246/f/uploads...
  - karmicthreat 6y ago
    Oh, you misunderstand. IT is an impediment to many control engineers. It's the automation techs and engineers that will work around the IT department if IT can't supply solutions. One of the more common ones being hide an AP or like in the article, use teamviewer or other remote access software. Then just share a common credential because nobody wants to actually pay for teamviewer. Businesses need lower cost because they are under price pressure. Especially with small utilities. Remote access is one of those ways to lower their costs on personnel or vendor support. There is still a whole lot of low hanging fruit in automation for improving security and access control. We're not going to get it from Rockwell for sure though.
- tsimionescu 6y ago
  Ideally, we should align our incentives such that having na internet-connected automation system is far more expensive than having one disconnected from the network. You should be forced by law to have a certain number of security experts on-call for any such system, periodic audits and pen-tests on your own expense etc. It's OK for a huge city operating many water treatment plants to decide that it is more efficient to automate and centralize and secure the network. It is horrendous that this is seen as the cheap solution for a small town.
  - uzakov 6y ago
    I agree with your comment but want to ask a couple of questions to see how you see it working it practice: What will stop the local city council be compliant on paper, ie them doing a tick box exercise and saying that their summer IT intern is the security department?
    - tsimionescu 6y ago
      I'm not a policy design expert by any means, and it's not like I've given this thorough thought. I expect some amount of red tape and controls from a government agency would be the proper way to enforce it. It would of course require significant political will to create these institutions and system of laws and regulations, but it could be similar in spirit to the kinds of controls the military has for software vendors that want to work with it.
EvanAnderson 6y ago
Yeah, but it's sadly common. I have personal experience with two such situations in my work over the last 16 years and I'm just some two-bit general IT contractor in Ohio, US. Until the decision makers who demand the interconnection of these networks are held accountable it's isn't going to stop. The cases I've seen have been to facilitate 24x7 off-site PLC vendor support access. I certainly see the business argument for the economics of off-site support for infrequently/improbably failure scenarios. At the very least, though, some type of physical interlock could have been employed (at the expense of some response time). Edit: I think controls can exist to make this kind of situation tenable for at least some types of industrial controls applications. When you start getting to things like municipal water and power I start getting more antsy.
ac29 6y ago
In a perfect world, maybe there would be unlimited budgets for small rural water districts to have 24/7 onsite staff and run highly secured networks. I regularly work with these sorts of water districts (larger, better funded ones as well). In reality, some of these small districts may only have 2 or 3 SCADA operators on staff. Sending them home with a pager, a tablet, a VPN password, and some overtime pay is a lot easier to get past the city council then taking on another two employees to cover the night shift for those rare events that need to be handled ASAP. I could share some real horror stories, but it wouldnt be professionally appropriate. Suffice to say, this story did not surprise me at all.
- nefitty 6y ago
  Of all the commentary I’ve read on this issue so far, this might be the scariest anecdote so far.
- joe_the_user 6y ago
  You parent is saying: "just never put your stuff in the Internet". I suppose it really is "that hard" to tell clients this and we can expect more problem down the line.
- rsync 6y ago
  "In a perfect world, maybe there would be unlimited budgets for small rural water districts to have 24/7 onsite staff and run highly secured networks." I reject this line of thought. A small rural water district can run with looser tolerances and looser guarantees - and have done so for decades. They should spend half the time (and a quarter of the money) setting up systems that fail safely and revert to known states and operate with looser tolerances. As for telemetry ... I am not joking at all when I say that a green light on the building that turns red and everyone in the county knows to call either Jed or Billy if that light is red is a completely reasonable system. It's a small rural water district (your words) after all, right ?
  - sillysaurusx 6y ago
    I invite you to become a pentester for a year and see if you still reject this line of thought.
renewiltord 6y ago
Sure, now you're paying 4x the price because you need two more operators so you can staff this place 100% of the time. And it's not even this guy. He probably reacts to like 4 different plants. Now you need like each of those places to have this guy in driving distance. You're not going to make it. Honestly, I think this is a fine outcome. There is a dollar value per life. I don't think we're undervaluing the life yet.
crystalmeph 6y ago
I work in industrial automation, and I agree. There’s constant rhetoric about buzzwords like “Industry 4.0,” which, if it means anything specifically, means “connect all the things.” There doesn’t seem to be a whole lot of thought around “is it even necessary for this three-ton industrial robot to be dynamically reprogrammed from a service center in Stockholm,” and it seems like everyone just assumes that everyone else will do a perfect job implementing and configuring security. I fear the tune will only change after the first multi-million dollar lawsuit, and I hope all that costs is the money.
- TeMPOraL 6y ago
  > There doesn’t seem to be a whole lot of thought around “is it even necessary for this three-ton industrial robot to be dynamically reprogrammed from a service center in Stockholm,” That's because the value proposition is only obvious when you substitute for "Stockholm" a city from one of the countries with cheaper labor. In my limited experience with Industry 4.0, it smells like a combination of forcing a goldrush to sell shovels (so many players that want to be the platform which connects everything) on one side, and ongoing search to turn capex into opex on the other (that latter thing is a trend in pretty much all industries, though). I think there's enough companies that would happily replace their control systems (and control engineers) with prepackaged control-as-a-service which they don't have to know anything about, supplied by the lowest bidder, to which they can shift any liability if anything happens. This kind of setup does require remote access.
  - p_l 6y ago
    The thing is, there is no "one size fits all" option most of the time. There are some packaged solutions but they all involve lots and lots of expert design, setup and management. So quite often it actually is for the interaction with possibly expensive engineer from random place in the world who has specific knowledge for the system involved, as well as enabling remote operations when facilities are in less accessible location. As for IIoT 4.0 - truth is a lot of industry was already heavily connected, and many functions I've seen so far are about getting deeper integration between ERP, MES, and individual work cells and workpiece tracking. Even when the workpiece is fried chicken waiting to be put in a bun, or a cut of pipe that will next need to be appropriately cleaned, bent, welded, painted and finally become part of a ship assembly.
vsareto 6y ago
>These "startups in the ICS space" are like turbotax/HRBlock: only continued idiocy allows their business model to exist. Nah, Dragos knows their shit. They'd be around even if ICS had good security.
- netflixandkill 6y ago
  We work with Dragos fairly regularly, they're solid. The main problem is that people who even consider the security or integrity of these systems are brought in years after they were specced, built, and more or less abandoned as built. The contractor and integrators then move on to the next project and copy what they did last time. Rinse , repeat. We've been actively pressing for realistic security and access control planning in the contract stage, but that's slow going in and of itself and still only affects new or upgraded installations -- on facilities with an expected lifetime of 10-30 years.
jacquesm 6y ago
Totally agreed. The reason why these systems are network connected is to save a few pennies on periodical drive-by's but they open up a whole can of worms in terms of risk that those entities are very ill equipped to deal with. The same was happening with SCADA systems for building management. Systems that were quite literally wide open were given an IPV4 address based on the assumption that since all they did was run HVAC controllers on obscure UDP ports that they were safe and nobody would bother with them.
netflixandkill 6y ago
Except everyone already does it and that toothpaste is never going back in the tube. We're fighting to keep these people from using unlicensed copies of TeamViewer for their primary access.
tgv 6y ago
> No cyber security products are needed. I'm fairly sure the Iranian ultracentrifuges were not connected, and were hacked anyway. Stuxnet was complicated, but being disconnected is not a 100% protection.
- rob74 6y ago
  Not 100%, but it takes orders of magnitude more motivation (and a nuclear program that would threaten your country if it were successful undoubtedly provides that) to accomplish...
  - tgv 6y ago
    Sure, but apart from low to medium effort hacking, cyber-warfare is still a possibility. Disabling infrastructure would be a high priority.
- uzakov 6y ago
  That is a slightly different Threat Model though, as well as(though I could be wrong on this one) the capabilities of attackers.
rootw0rm 6y ago
best comment i've read in a while. KISS
72deluxe 6y ago
Open your modbus port on your server and see how often you get hit! Then look for a list of open modbus ports on the Internet and be wowed at all the industrial machinery that is just sat on the Internet...
JoeAltmaier 6y ago
Yet on the internet there are astronomical levels of 'griefers', people who just want to see the world burn. The internet magnifies this tremendously. The internet offers the appearance of anonymity. This is a dangerous combination. Drunk drivers are in the accident too. They take damage. Not so for a hack. In real life, the internals of a water plant are behind locked doors. Not everybody from Nairobi to Nantucket can get in and do as they like. I'm afraid that trust in the public is definitely not the way to go with infrastructure and networked control systems.
johnwyles 6y ago
Odd to pick HRBlock/Intuit as they are used for convenience and because we don't have a central system like Sweden and have in the US 50 different states and territories with 50+ different tax rules and so many edge cases in existence it's almost the rule than the exception each individual has one. Likewise these consultants are not just coming in and pointing their fingers at the obvious ICS on the internet. They are also providing services to understand why they were attached in the first place and where that process broke down, how to keep the current ease with which to operate the system, and implement the transitions.

by rsync 6y ago

deleted 6y ago

[deleted]

karmicthreat 6y ago

I think these days it is becoming a business need though. These systems are made by vendors who probably need remote access. Also if the plant has a relatively unsophisticated IT department then someone is just going to shove an AP in the ceiling so they can check things when they get called at 1AM. Several ICS vendors like Tosibox and EWON make devices to accomplish this. I think Tosi has the more secure model, though I hate their proprietary dongles. VPNs are also used pretty successfully here. Several large companies also don't let you directly connect to anything. You vpn in and connect to a machine with Citrix and then you can use whatever was setup for your there. Usually whatever version of Logix/Studio 5000 the plant is on. You have to talk to someone in IT to get your files moved in/out. I think Amazon went a different direction and uses Versiondog to monitor their automation systems and check for changes. I don't work there or know anyone on their automation team so I'm not aware of the details. Still, I think you can have external access and be secure. You just need to balance things out with your business needs.

EvanAnderson 6y ago
VPNs are also used pretty successfully here. Several large companies also don't let you directly connect to anything. You vpn in and connect to a machine with Citrix and then you can use whatever was setup for your there. Usually whatever version of Logix/Studio 5000 the plant is on. I support an environment almost exactly like this (albeit in a small manufacturing company). I don't love having one of the controls networks attached, in any way, to the LAN, but I understand the business requirements justify it. It happens that there's a controls system running devices that could cause massive environmental impact in some malfunction scenarios. I am happy to report the plant, being held to account for things like public evacuation plans and hazmat filings with local first responders, has never asked about connecting that network to anything. That would be a walk-out-the-door type scenario for me. I worry that they'd just find somebody who wouldn't have those scruples, though.
- just_steve_h 6y ago
  > I worry that they'd just find somebody who wouldn't have those scruples, though. If I may speak slightly out of turn to a stranger, using a possible & currently imagined future person less scrupulous does not modify in any way your obligation, however you perceive it, to act ethically.
  - EvanAnderson 6y ago
    I've warned them about my concerns. If I stop working with them, and I have no further knowledge of thir situation, I don't see what else there would be for me to do.
  - Dylan16807 6y ago
    Ohhhhkay? Nobody was suggesting that it does. The person you're replying to specifically said the opposite.
- karmicthreat 6y ago
  That's an understandable scenario and good on you guys for balancing the risks.
- uzakov 6y ago
  What are your thoughts on something like this in your line of work? https://cloud.google.com/beyondcorp https://cloud.google.com/beyondcorp
  - EvanAnderson 6y ago
    I may be too much of a simple "IT guy" to grok the deep meaning of BeyondCorp. I read thru some of the various papers when they came out and always came back to the thought "Yeah, that's nice if you have the resources to exert control over that much of your technology stack." I don't have those resources, nor do my Customers. I've got the various mix of Windows, Linux, and embedded devices that the Customer has purchased to serve their business applications. They (and I) don't have the clout or purchasing power to demand application vendors bend to our desires, so I'm left with making the best out of sub-optimal architecture, protocols, etc. Google says, in the BeyondCorp III paper under the heading "Third-Party Software"[1]: Third-party software has frequently proved troublesome, as sometimes it can’t present TLS certificates, and sometimes it assumes direct connectivity. In order to support these tools, we developed a solution to automatically establish encrypted point-to-point tunnels (using a TUN device). The software is unaware of the tunnel, and behaves as if it’s directly connected to the server. So, they just do what I do and throw a VPN at it, albeit a client-to-server VPN serving an individual application rather than a client-to-network VPN like I might. I do my best to segment the networks at my Customer sites, to use default-deny policies between security zones, to authenticate traffic flows to users and devices where possible, and when unable (because of limitations of client software/devices, usually) restrict access by source address. Within each security zone I try to make a worst-case assumption of an attacker getting complete access to the zone (compromising a host within the zone and getting arbitrary network access, for example) with things like private VLANs and host-based firewalls. I have to declare "bankruptcy" in some security zones (usually where there are embedded devices) where I have to rely only on network segmentation because the devices (or vendors) are too "stupid" to have host-based firewall functionality, authentication, encryption, etc. (These are the devices that fall over and die when they get port-scanned, yet somehow end up in mission-critical roles.) I think the harsh reality is that, operating at the scale of small to mid-sized companies, IT and infosec are forced into a lot of bad places by vendors who don't care, and management who are focused on the bottom-line and who don't see security as anything other than something to purchase insurance for. To put it another way: I have to make all this crap work. If I make it too difficult for the end users to work or for the vendors to support I'll be kicked to the curb and they'll find somebody else who will be less "difficult". [1] https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45728.pdf https://storage.googleapis.com/pub-tools-public-publication-...
closeparen 6y ago
Remote Desktop was exactly the mechanism here, the attacker used TeamViewer to work the UI on a plant operator’s desktop and he happened to be watching.
- rootsudo 6y ago
  Seeing reports now that it was indeed Teamviewer! They also have a satellite office over in Clearwater, Florida (which is trying to be like a little bay area copy, v2/3) Interesting, but, Teamviewer also has been exploited and leaked creds, but took three years to confirm it: https://www.bleepingcomputer.com/news/security/teamviewer-confirms-undisclosed-breach-from-2016/ https://www.bleepingcomputer.com/news/security/teamviewer-co... Or, if the client computer browsed a site, it'd actually start open an SMB share on the perps computer: https://www.bleepingcomputer.com/news/security/teamviewer-fixes-bug-that-lets-attackers-access-your-pc/ https://www.bleepingcomputer.com/news/security/teamviewer-fi... and a few other interesting vulnerabilities, hmm.
- karmicthreat 6y ago
  Citrix is a little more than teamviewer. And can be encapsulated in a VPN as well. Teamviewer on a desktop, probably with a shared credential isn't very secure. Knowing this though, I doubt that it was a teamviewer exploit. My guess would be a disgruntled employee since they knew what to get into to change chemical set points.
  - closeparen 6y ago
    They both expose Remote Desktop to the internet given the proper credentials, and I’m guessing here but I think it’s pretty likely that the attacker had credentials. Whether it was a disgruntled insider, a dumb password, or (most likely) a reused password from a leak somewhere. I would be interested to know if the TeamViewer account in question had 2FA... probably not.
Veserv 6y ago
It is not a business "need". These systems have functioned without remote access perfectly well for decades. It is a business "want" and thus must be balanced against any new risks relative to historical risks. The risk of adding remote access to critical systems is the introduction of globally accessible single-point-of-failures. Given the nature of software, such an attack has an unlimited amount of time to be perfected before deployment and when finished can be deployed at effectively zero cost and complete in effectively zero time which provides no meaningful way to respond except with already deployed automated systems. So, the risk added with remote access is the risk of malicious catastrophic total system failure. In this case, the water treatment facility treated the water for ~15,000 residents. In a similar case many years ago [1], a similar event occurred to a water treatment facility that treated the water for ~12,000 residents which resulted in 100 affected individuals before the effects were detected. So, we can reasonably assume that undetected water treatment tampering on a facility serving ~10,000 individuals will result in about ~100 affected individuals before the effects are detected. If there exists a way to tamper with a water treatment facility that would result in deaths for the affected individuals, which is quite likely, then that means the risk of remote access to the water treatment facility is ~100 deaths. So, as a society, we should ask the question: What is the standard of care that should be applied to a system where failure may result in the deaths of 100 people? And any business that wishes to add remote access to such a system must demonstrate to the satisfaction of society that they are taking that degree of care. It is not the role of society or the people to suffer for the convenience of business. And in this case, I am certain that they are not taking an appropriate amount of care. The fact that you honestly suggested that an IT department would shove an AP in the ceiling for their convenience shows just how low our expectations are. In any other industry, such an act would be, in no uncertain terms, criminal negligence. That our standard assumption about the standard of care taken is criminal negligence shows just how far any of these companies is from actually deploying systems that have external access and have adequate security. [1] https://www.spencerma.gov/sites/g/files/vyhlif1246/f/uploads/wateremergency.pdf https://www.spencerma.gov/sites/g/files/vyhlif1246/f/uploads...
- karmicthreat 6y ago
  Oh, you misunderstand. IT is an impediment to many control engineers. It's the automation techs and engineers that will work around the IT department if IT can't supply solutions. One of the more common ones being hide an AP or like in the article, use teamviewer or other remote access software. Then just share a common credential because nobody wants to actually pay for teamviewer. Businesses need lower cost because they are under price pressure. Especially with small utilities. Remote access is one of those ways to lower their costs on personnel or vendor support. There is still a whole lot of low hanging fruit in automation for improving security and access control. We're not going to get it from Rockwell for sure though.
tsimionescu 6y ago
Ideally, we should align our incentives such that having na internet-connected automation system is far more expensive than having one disconnected from the network. You should be forced by law to have a certain number of security experts on-call for any such system, periodic audits and pen-tests on your own expense etc. It's OK for a huge city operating many water treatment plants to decide that it is more efficient to automate and centralize and secure the network. It is horrendous that this is seen as the cheap solution for a small town.
- uzakov 6y ago
  I agree with your comment but want to ask a couple of questions to see how you see it working it practice: What will stop the local city council be compliant on paper, ie them doing a tick box exercise and saying that their summer IT intern is the security department?
  - tsimionescu 6y ago
    I'm not a policy design expert by any means, and it's not like I've given this thorough thought. I expect some amount of red tape and controls from a government agency would be the proper way to enforce it. It would of course require significant political will to create these institutions and system of laws and regulations, but it could be similar in spirit to the kinds of controls the military has for software vendors that want to work with it.

EvanAnderson 6y ago

Yeah, but it's sadly common. I have personal experience with two such situations in my work over the last 16 years and I'm just some two-bit general IT contractor in Ohio, US. Until the decision makers who demand the interconnection of these networks are held accountable it's isn't going to stop. The cases I've seen have been to facilitate 24x7 off-site PLC vendor support access. I certainly see the business argument for the economics of off-site support for infrequently/improbably failure scenarios. At the very least, though, some type of physical interlock could have been employed (at the expense of some response time). Edit: I think controls can exist to make this kind of situation tenable for at least some types of industrial controls applications. When you start getting to things like municipal water and power I start getting more antsy.

ac29 6y ago

In a perfect world, maybe there would be unlimited budgets for small rural water districts to have 24/7 onsite staff and run highly secured networks. I regularly work with these sorts of water districts (larger, better funded ones as well). In reality, some of these small districts may only have 2 or 3 SCADA operators on staff. Sending them home with a pager, a tablet, a VPN password, and some overtime pay is a lot easier to get past the city council then taking on another two employees to cover the night shift for those rare events that need to be handled ASAP. I could share some real horror stories, but it wouldnt be professionally appropriate. Suffice to say, this story did not surprise me at all.

nefitty 6y ago
Of all the commentary I’ve read on this issue so far, this might be the scariest anecdote so far.
joe_the_user 6y ago
You parent is saying: "just never put your stuff in the Internet". I suppose it really is "that hard" to tell clients this and we can expect more problem down the line.
rsync 6y ago
"In a perfect world, maybe there would be unlimited budgets for small rural water districts to have 24/7 onsite staff and run highly secured networks." I reject this line of thought. A small rural water district can run with looser tolerances and looser guarantees - and have done so for decades. They should spend half the time (and a quarter of the money) setting up systems that fail safely and revert to known states and operate with looser tolerances. As for telemetry ... I am not joking at all when I say that a green light on the building that turns red and everyone in the county knows to call either Jed or Billy if that light is red is a completely reasonable system. It's a small rural water district (your words) after all, right ?
- sillysaurusx 6y ago
  I invite you to become a pentester for a year and see if you still reject this line of thought.

renewiltord 6y ago

Sure, now you're paying 4x the price because you need two more operators so you can staff this place 100% of the time. And it's not even this guy. He probably reacts to like 4 different plants. Now you need like each of those places to have this guy in driving distance. You're not going to make it. Honestly, I think this is a fine outcome. There is a dollar value per life. I don't think we're undervaluing the life yet.

crystalmeph 6y ago

I work in industrial automation, and I agree. There’s constant rhetoric about buzzwords like “Industry 4.0,” which, if it means anything specifically, means “connect all the things.” There doesn’t seem to be a whole lot of thought around “is it even necessary for this three-ton industrial robot to be dynamically reprogrammed from a service center in Stockholm,” and it seems like everyone just assumes that everyone else will do a perfect job implementing and configuring security. I fear the tune will only change after the first multi-million dollar lawsuit, and I hope all that costs is the money.

TeMPOraL 6y ago
> There doesn’t seem to be a whole lot of thought around “is it even necessary for this three-ton industrial robot to be dynamically reprogrammed from a service center in Stockholm,” That's because the value proposition is only obvious when you substitute for "Stockholm" a city from one of the countries with cheaper labor. In my limited experience with Industry 4.0, it smells like a combination of forcing a goldrush to sell shovels (so many players that want to be the platform which connects everything) on one side, and ongoing search to turn capex into opex on the other (that latter thing is a trend in pretty much all industries, though). I think there's enough companies that would happily replace their control systems (and control engineers) with prepackaged control-as-a-service which they don't have to know anything about, supplied by the lowest bidder, to which they can shift any liability if anything happens. This kind of setup does require remote access.
- p_l 6y ago
  The thing is, there is no "one size fits all" option most of the time. There are some packaged solutions but they all involve lots and lots of expert design, setup and management. So quite often it actually is for the interaction with possibly expensive engineer from random place in the world who has specific knowledge for the system involved, as well as enabling remote operations when facilities are in less accessible location. As for IIoT 4.0 - truth is a lot of industry was already heavily connected, and many functions I've seen so far are about getting deeper integration between ERP, MES, and individual work cells and workpiece tracking. Even when the workpiece is fried chicken waiting to be put in a bun, or a cut of pipe that will next need to be appropriately cleaned, bent, welded, painted and finally become part of a ship assembly.

vsareto 6y ago

>These "startups in the ICS space" are like turbotax/HRBlock: only continued idiocy allows their business model to exist. Nah, Dragos knows their shit. They'd be around even if ICS had good security.

netflixandkill 6y ago
We work with Dragos fairly regularly, they're solid. The main problem is that people who even consider the security or integrity of these systems are brought in years after they were specced, built, and more or less abandoned as built. The contractor and integrators then move on to the next project and copy what they did last time. Rinse , repeat. We've been actively pressing for realistic security and access control planning in the contract stage, but that's slow going in and of itself and still only affects new or upgraded installations -- on facilities with an expected lifetime of 10-30 years.

jacquesm 6y ago

Totally agreed. The reason why these systems are network connected is to save a few pennies on periodical drive-by's but they open up a whole can of worms in terms of risk that those entities are very ill equipped to deal with. The same was happening with SCADA systems for building management. Systems that were quite literally wide open were given an IPV4 address based on the assumption that since all they did was run HVAC controllers on obscure UDP ports that they were safe and nobody would bother with them.

netflixandkill 6y ago

Except everyone already does it and that toothpaste is never going back in the tube. We're fighting to keep these people from using unlicensed copies of TeamViewer for their primary access.

tgv 6y ago

> No cyber security products are needed. I'm fairly sure the Iranian ultracentrifuges were not connected, and were hacked anyway. Stuxnet was complicated, but being disconnected is not a 100% protection.

rob74 6y ago
Not 100%, but it takes orders of magnitude more motivation (and a nuclear program that would threaten your country if it were successful undoubtedly provides that) to accomplish...
- tgv 6y ago
  Sure, but apart from low to medium effort hacking, cyber-warfare is still a possibility. Disabling infrastructure would be a high priority.
uzakov 6y ago
That is a slightly different Threat Model though, as well as(though I could be wrong on this one) the capabilities of attackers.

rootw0rm 6y ago

best comment i've read in a while. KISS

72deluxe 6y ago

Open your modbus port on your server and see how often you get hit! Then look for a list of open modbus ports on the Internet and be wowed at all the industrial machinery that is just sat on the Internet...

JoeAltmaier 6y ago

Yet on the internet there are astronomical levels of 'griefers', people who just want to see the world burn. The internet magnifies this tremendously. The internet offers the appearance of anonymity. This is a dangerous combination. Drunk drivers are in the accident too. They take damage. Not so for a hack. In real life, the internals of a water plant are behind locked doors. Not everybody from Nairobi to Nantucket can get in and do as they like. I'm afraid that trust in the public is definitely not the way to go with infrastructure and networked control systems.

johnwyles 6y ago

Odd to pick HRBlock/Intuit as they are used for convenience and because we don't have a central system like Sweden and have in the US 50 different states and territories with 50+ different tax rules and so many edge cases in existence it's almost the rule than the exception each individual has one. Likewise these consultants are not just coming in and pointing their fingers at the obvious ICS on the internet. They are also providing services to understand why they were attached in the first place and where that process broke down, how to keep the current ease with which to operate the system, and implement the transitions.