11 ms·

Internet-accessble industrial control systems have been a problem for many years now. It's a documented issue but it's difficult to fix for a variety of reasons

Internet-accessble industrial control systems have been a problem for many years now. It's a documented issue but it's difficult to fix for a variety of reasons: 1. Difficult to identify the owner: a lot of the devices are on mobile networks that don't point to an obvious owner. 2. Unknown criticality: is it a demo system or something used in production? 3. Security budget: lots of smaller utilities don't have a budget for buying cyber security products. 4. Uneducated vendor: sometimes the vendors of the device give very bad advice (https://blog.shodan.io/why-control-systems-are-on-the-internet/ https://blog.shodan.io/why-control-systems-are-on-the-intern...) That being said, based on the numbers in Shodan the situation has improved over the past decade. And there's been a large resurgence of startups in the ICS space (ex https://www.dragos.com https://www.dragos.com, https://www.gravwell.io https://www.gravwell.io). Here's a current view of exposed industrial devices on the Internet: https://beta.shodan.io/search/report?query=tag%3Aics&title=Industrial%20Control%20Systems%20Overview https://beta.shodan.io/search/report?query=tag%3Aics&title=I... I've written/ presented on the issue a few times: https://blog.shodan.io/taking-things-offline-is-hard/ https://blog.shodan.io/taking-things-offline-is-hard/ https://blog.shodan.io/trends-in-internet-exposure/ https://blog.shodan.io/trends-in-internet-exposure/ https://exposure.shodan.io/#/ https://exposure.shodan.io/#/

giantg2 6y ago
Even non-connected systems can be a problem. Stuxnet was an example. But I think the main point is that owners of those systems think they are protected just by being disconnected.
- nopzor 6y ago
  the stuxnet attack had a significantly higher level of sophistication than this. if your threat is a competent nation state the bar is much much higher.
  - just_steve_h 6y ago
    Is the threat not from a competent nation-state or supranational entity? Is that not the intention of designating power & water & electricity systems as "critical infrastructure?"
    - dodobirdlord 6y ago
      I don’t think the water treatment plant of Oldsmar, Florida falls under the same threat model as a uranium enrichment facility.
      - tapland 6y ago
        I don't think the threat is much different, but an attacker doesn't have the patience for a stuxnet level attack on one of many water treatment facilities. If it was airgapped it wouldn't be available to easily be used by a nation state to attack infrastructure in case of other simultaneous attacks.
  - giantg2 6y ago
    Remember the concerns during the first gulf war about Iranians potentially planning to contaminate drinking water in the US?
- sellyme 6y ago
  > Stuxnet Yeah, and even the best security practices aren't going to work too well if someone drops a nuke on your facility. Stuxnet was an extraordinarily sophisticated attack well beyond what a typical industrial system will need to protect against, or even be able to protect against. It's not really in the same league as anyone being able to just remote in and change settings, and while it's realistic to expect a bloke called Steve who runs the computers at the water processing plant to prevent someone just remoting in willy-nilly, it's not as realistic to expect him to defend against two nation-states working together explicitly targeting that facility.
  - cm2187 6y ago
    Would have been better with “homer” than “steve”!
  - giantg2 6y ago
    "...well beyond what a typical industrial system will need to protect against..." And the target system also had security systems well beyond the capability of your local water treatment plant. Let's not forget that these assets deemed as critical infrastructure could be the target of nation states. All I'm saying is that not being connected is only a small part of security for industrial systems, and that some people wrongly rely on it being enough.
    - yodelshady 6y ago
      The target system probably fared better than you think - as a whole, it certainly wasn't destroyed. As with this incident, operators were physically present. That seems to be the real lesson (even if - see other war stories on this thread - operators tend to themselves have a creative approach to network security).
      - ClumsyPilot 6y ago
        The issue was not identified for months, and from what I understood, a significan fraction of the certrifuges were destroyed. The attack worked exactly as designed - wasting time, destoying equipment while being stealthy. Had the attack tried to destoroy all equipment at once" it would have beem spotted immediately.
the_only_law 6y ago
There’s probably a dumb reason I’m not thinking of, but why does the US have such a higher count than other large, industrialized nations?
- rootsudo 6y ago
  Underpaid IT/Infosec. People conflate IT and Infosec, once it's on an Govt payroll for billing purposes, no one touches the system if it's on a network provider, and not internal. If not internal, it won't show up on audits, most IT departments deal with a Windows Domain/Network and that's most locked down, but if it doesn't share a true connection physically, it's exempted from most audits. The question is, why are the telecom providers allowing this, but there's also alot of legacy stuff they don't want to touch as it may violate the terms/contract and the bandwidth isn't the issue, so telecoms largely ignore it as they're just a bridge/
- achillean 6y ago
  Some mobile networks in the US will give you a public IP whereas in most other countries they do Carrier-NAT. You can get a better sense of it when looking at the IP space owners for the devices: https://beta.shodan.io/search/facet?query=tag%3Aics+country%3AUS&facet=org https://beta.shodan.io/search/facet?query=tag%3Aics+country%...
- jcranmer 6y ago
  The US is a much larger country than countries like Germany or France. If you add up a roughly equal-sized amount of the European Union for comparison, you get a number of hosts around 30k-ish, which is somewhat lower than the US's 34-35k, but not by all that much.
  - the_only_law 6y ago
    I’m not talking about European countries. I’m looking at countries like China, Russia, Brazil, India, etc.
    - belval 6y ago
rsync 6y ago
"Internet-accessble industrial control systems have been a problem for many years now ..." They are a problem the way drunk driving is a problem. You just don't ever do it. Ever. No cyber security products are needed. No budget required. These "startups in the ICS space" are like turbotax/HRBlock: only continued idiocy allows their business model to exist.
- deleted 6y ago
  [deleted]
- karmicthreat 6y ago
  I think these days it is becoming a business need though. These systems are made by vendors who probably need remote access. Also if the plant has a relatively unsophisticated IT department then someone is just going to shove an AP in the ceiling so they can check things when they get called at 1AM. Several ICS vendors like Tosibox and EWON make devices to accomplish this. I think Tosi has the more secure model, though I hate their proprietary dongles. VPNs are also used pretty successfully here. Several large companies also don't let you directly connect to anything. You vpn in and connect to a machine with Citrix and then you can use whatever was setup for your there. Usually whatever version of Logix/Studio 5000 the plant is on. You have to talk to someone in IT to get your files moved in/out. I think Amazon went a different direction and uses Versiondog to monitor their automation systems and check for changes. I don't work there or know anyone on their automation team so I'm not aware of the details. Still, I think you can have external access and be secure. You just need to balance things out with your business needs.
  - EvanAnderson 6y ago
    VPNs are also used pretty successfully here. Several large companies also don't let you directly connect to anything. You vpn in and connect to a machine with Citrix and then you can use whatever was setup for your there. Usually whatever version of Logix/Studio 5000 the plant is on. I support an environment almost exactly like this (albeit in a small manufacturing company). I don't love having one of the controls networks attached, in any way, to the LAN, but I understand the business requirements justify it. It happens that there's a controls system running devices that could cause massive environmental impact in some malfunction scenarios. I am happy to report the plant, being held to account for things like public evacuation plans and hazmat filings with local first responders, has never asked about connecting that network to anything. That would be a walk-out-the-door type scenario for me. I worry that they'd just find somebody who wouldn't have those scruples, though.
cookiengineer 6y ago
5. Plausible deniability. I wanted to add this point, because lack of security measurements and the convenient existence of hackers allows a company plausible deniability. As long as companies are not legally forced to take precautious security measurements, they won't. And it plays into their advantage, because insurance providers have rarely clauses in them regarding minimum security measurements.
tgv 6y ago
> 1. Difficult to identify the owner That's not necessary: just make the board of directors of the companies that operate it, have it on their premises, or use it personally responsible. That should give them more than enough incentive.
HenryBemis 6y ago
From personal experience. I was working in a factory producing food (sorry not saying what type). The "machine" producing and packaging the food was a huge 20m by 3m by 3m metal box (imagine a bus). One end - raw material & packaging goes in, far end, packaged food comes out nice and neat. That machine's interface was a Windows 95 (YES!) German language version. I am not German. I do not speak/read/write German. It was in that factory's IT admin & support. Nobody in that factory's operations staff could read German. So the rule is "we never touch this machine - never EVER. Anything that goes wrong (sounds, visuals, etc.) we ring the bell, escalate, get the vendor in." Sidenote: For the youngsters, W95 was an OS by Microsoft, before you were born, and it did not have the multi-use/control environment (admin = god, user = cannot install sofrware, etc.) The machine had two 'terminals'. One ON the machine (physically - on the front of the 'bus') and one 'remote' (50m away) in an office, with a huge window where you could observe the machine. Both screens displayed the exact same desktop (Win95, German)(basically a single computer with two monitors 'duplicating'. Geniuses operation staff got bored looking at a machine with no error/faults (German built!) and installed a software that came along with adult video CDs (we're talking early 00's). Geniuses were watching porn on a machine that was worth many millions and was the production machine. When the geniuses were watching porn, it was being displayed in BOTH screens. Factory floor, AND office 50m away. Sound and all... So.. adding to your points: 5. System limitations and customisation/hardening (no need for extra software - just basic security hardening/configuration)(win95) 6. Uneducated users (employers installing video player from an adult video CD)
sschueller 6y ago
There was this website a while back called "vnc roulette". It would randomly connect you to a open VNC host. Many of those where control systems all over the world.
isbvhodnvemrwvn 6y ago
They don't even need to be internet-accessible, physical security is often weak as well. Surprisingly relevant: https://youtu.be/rnmcRTnTNC8?t=252 https://youtu.be/rnmcRTnTNC8?t=252
bottled_poe 6y ago
This explanation has a lot of good reasons, but is missing an important one - the value proposition of cyber security. Decisions makers (assuming informed) will make an assessment of risk vs cost. Absolute cyber security is rarely a relevant consideration. The assessment is always going to be (at best) an evaluation of investment in cyber security vs risk of greater costs (in the form of compromised security, organisational changes, etc). We need to understand that these decisions are not made from a purely technical perspective. Real costs exist and decision-makers will (rightly) always compare those costs against the estimated benefits.
- dreamcompiler 6y ago
  And because it's an expenditure that only hypothetically might decrease a larger expenditure in the future, many managers will decide to do only the minimum necessary to check the compliance boxes. Another problem is that when the security systems get in the way of expediency, there's always somebody around who can disable or severely cripple the security to make it easy for people to e.g. work from home during a pandemic.
  - bottled_poe 6y ago
    I think the economics of cyber security are poorly modelled/understood at present. I’m of the opinion that building a slightly higher wall than a similar target is generally sufficient (as an economic deterrent) vs most enemies. However, this is a simplistic model and doesn’t account for targeted attacks. It’s a complex problem space and has a lot of room to mature. I expect great changes in this space over the coming years.
Kim_Bruning 6y ago
Wrt uneducated vendors: Industrial Control systems tend to be built by people with an electrical background rather than an IT background, and they have their own culture, and strong Not Invented Here effect. A strong cross-disciplinary startup could make a killing in industrial automation. (And extant companies that remotely meet that criterion already seem to do so.)

giantg2 6y ago

Even non-connected systems can be a problem. Stuxnet was an example. But I think the main point is that owners of those systems think they are protected just by being disconnected.

nopzor 6y ago
the stuxnet attack had a significantly higher level of sophistication than this. if your threat is a competent nation state the bar is much much higher.
- just_steve_h 6y ago
  Is the threat not from a competent nation-state or supranational entity? Is that not the intention of designating power & water & electricity systems as "critical infrastructure?"
  - dodobirdlord 6y ago
    I don’t think the water treatment plant of Oldsmar, Florida falls under the same threat model as a uranium enrichment facility.
    - tapland 6y ago
      I don't think the threat is much different, but an attacker doesn't have the patience for a stuxnet level attack on one of many water treatment facilities. If it was airgapped it wouldn't be available to easily be used by a nation state to attack infrastructure in case of other simultaneous attacks.
- giantg2 6y ago
  Remember the concerns during the first gulf war about Iranians potentially planning to contaminate drinking water in the US?
sellyme 6y ago
> Stuxnet Yeah, and even the best security practices aren't going to work too well if someone drops a nuke on your facility. Stuxnet was an extraordinarily sophisticated attack well beyond what a typical industrial system will need to protect against, or even be able to protect against. It's not really in the same league as anyone being able to just remote in and change settings, and while it's realistic to expect a bloke called Steve who runs the computers at the water processing plant to prevent someone just remoting in willy-nilly, it's not as realistic to expect him to defend against two nation-states working together explicitly targeting that facility.
- cm2187 6y ago
  Would have been better with “homer” than “steve”!
- giantg2 6y ago
  "...well beyond what a typical industrial system will need to protect against..." And the target system also had security systems well beyond the capability of your local water treatment plant. Let's not forget that these assets deemed as critical infrastructure could be the target of nation states. All I'm saying is that not being connected is only a small part of security for industrial systems, and that some people wrongly rely on it being enough.
  - yodelshady 6y ago
    The target system probably fared better than you think - as a whole, it certainly wasn't destroyed. As with this incident, operators were physically present. That seems to be the real lesson (even if - see other war stories on this thread - operators tend to themselves have a creative approach to network security).
    - ClumsyPilot 6y ago
      The issue was not identified for months, and from what I understood, a significan fraction of the certrifuges were destroyed. The attack worked exactly as designed - wasting time, destoying equipment while being stealthy. Had the attack tried to destoroy all equipment at once" it would have beem spotted immediately.

the_only_law 6y ago

There’s probably a dumb reason I’m not thinking of, but why does the US have such a higher count than other large, industrialized nations?

rootsudo 6y ago
Underpaid IT/Infosec. People conflate IT and Infosec, once it's on an Govt payroll for billing purposes, no one touches the system if it's on a network provider, and not internal. If not internal, it won't show up on audits, most IT departments deal with a Windows Domain/Network and that's most locked down, but if it doesn't share a true connection physically, it's exempted from most audits. The question is, why are the telecom providers allowing this, but there's also alot of legacy stuff they don't want to touch as it may violate the terms/contract and the bandwidth isn't the issue, so telecoms largely ignore it as they're just a bridge/
achillean 6y ago
Some mobile networks in the US will give you a public IP whereas in most other countries they do Carrier-NAT. You can get a better sense of it when looking at the IP space owners for the devices: https://beta.shodan.io/search/facet?query=tag%3Aics+country%3AUS&facet=org https://beta.shodan.io/search/facet?query=tag%3Aics+country%...
jcranmer 6y ago
The US is a much larger country than countries like Germany or France. If you add up a roughly equal-sized amount of the European Union for comparison, you get a number of hosts around 30k-ish, which is somewhat lower than the US's 34-35k, but not by all that much.
- the_only_law 6y ago
  I’m not talking about European countries. I’m looking at countries like China, Russia, Brazil, India, etc.
  - belval 6y ago

rsync 6y ago

"Internet-accessble industrial control systems have been a problem for many years now ..." They are a problem the way drunk driving is a problem. You just don't ever do it. Ever. No cyber security products are needed. No budget required. These "startups in the ICS space" are like turbotax/HRBlock: only continued idiocy allows their business model to exist.

deleted 6y ago
[deleted]
karmicthreat 6y ago
I think these days it is becoming a business need though. These systems are made by vendors who probably need remote access. Also if the plant has a relatively unsophisticated IT department then someone is just going to shove an AP in the ceiling so they can check things when they get called at 1AM. Several ICS vendors like Tosibox and EWON make devices to accomplish this. I think Tosi has the more secure model, though I hate their proprietary dongles. VPNs are also used pretty successfully here. Several large companies also don't let you directly connect to anything. You vpn in and connect to a machine with Citrix and then you can use whatever was setup for your there. Usually whatever version of Logix/Studio 5000 the plant is on. You have to talk to someone in IT to get your files moved in/out. I think Amazon went a different direction and uses Versiondog to monitor their automation systems and check for changes. I don't work there or know anyone on their automation team so I'm not aware of the details. Still, I think you can have external access and be secure. You just need to balance things out with your business needs.
- EvanAnderson 6y ago
  VPNs are also used pretty successfully here. Several large companies also don't let you directly connect to anything. You vpn in and connect to a machine with Citrix and then you can use whatever was setup for your there. Usually whatever version of Logix/Studio 5000 the plant is on. I support an environment almost exactly like this (albeit in a small manufacturing company). I don't love having one of the controls networks attached, in any way, to the LAN, but I understand the business requirements justify it. It happens that there's a controls system running devices that could cause massive environmental impact in some malfunction scenarios. I am happy to report the plant, being held to account for things like public evacuation plans and hazmat filings with local first responders, has never asked about connecting that network to anything. That would be a walk-out-the-door type scenario for me. I worry that they'd just find somebody who wouldn't have those scruples, though.

cookiengineer 6y ago

5. Plausible deniability. I wanted to add this point, because lack of security measurements and the convenient existence of hackers allows a company plausible deniability. As long as companies are not legally forced to take precautious security measurements, they won't. And it plays into their advantage, because insurance providers have rarely clauses in them regarding minimum security measurements.

tgv 6y ago

> 1. Difficult to identify the owner That's not necessary: just make the board of directors of the companies that operate it, have it on their premises, or use it personally responsible. That should give them more than enough incentive.

HenryBemis 6y ago

From personal experience. I was working in a factory producing food (sorry not saying what type). The "machine" producing and packaging the food was a huge 20m by 3m by 3m metal box (imagine a bus). One end - raw material & packaging goes in, far end, packaged food comes out nice and neat. That machine's interface was a Windows 95 (YES!) German language version. I am not German. I do not speak/read/write German. It was in that factory's IT admin & support. Nobody in that factory's operations staff could read German. So the rule is "we never touch this machine - never EVER. Anything that goes wrong (sounds, visuals, etc.) we ring the bell, escalate, get the vendor in." Sidenote: For the youngsters, W95 was an OS by Microsoft, before you were born, and it did not have the multi-use/control environment (admin = god, user = cannot install sofrware, etc.) The machine had two 'terminals'. One ON the machine (physically - on the front of the 'bus') and one 'remote' (50m away) in an office, with a huge window where you could observe the machine. Both screens displayed the exact same desktop (Win95, German)(basically a single computer with two monitors 'duplicating'. Geniuses operation staff got bored looking at a machine with no error/faults (German built!) and installed a software that came along with adult video CDs (we're talking early 00's). Geniuses were watching porn on a machine that was worth many millions and was the production machine. When the geniuses were watching porn, it was being displayed in BOTH screens. Factory floor, AND office 50m away. Sound and all... So.. adding to your points: 5. System limitations and customisation/hardening (no need for extra software - just basic security hardening/configuration)(win95) 6. Uneducated users (employers installing video player from an adult video CD)

sschueller 6y ago

There was this website a while back called "vnc roulette". It would randomly connect you to a open VNC host. Many of those where control systems all over the world.

isbvhodnvemrwvn 6y ago

They don't even need to be internet-accessible, physical security is often weak as well. Surprisingly relevant: https://youtu.be/rnmcRTnTNC8?t=252 https://youtu.be/rnmcRTnTNC8?t=252

bottled_poe 6y ago

This explanation has a lot of good reasons, but is missing an important one - the value proposition of cyber security. Decisions makers (assuming informed) will make an assessment of risk vs cost. Absolute cyber security is rarely a relevant consideration. The assessment is always going to be (at best) an evaluation of investment in cyber security vs risk of greater costs (in the form of compromised security, organisational changes, etc). We need to understand that these decisions are not made from a purely technical perspective. Real costs exist and decision-makers will (rightly) always compare those costs against the estimated benefits.

dreamcompiler 6y ago
And because it's an expenditure that only hypothetically might decrease a larger expenditure in the future, many managers will decide to do only the minimum necessary to check the compliance boxes. Another problem is that when the security systems get in the way of expediency, there's always somebody around who can disable or severely cripple the security to make it easy for people to e.g. work from home during a pandemic.
- bottled_poe 6y ago
  I think the economics of cyber security are poorly modelled/understood at present. I’m of the opinion that building a slightly higher wall than a similar target is generally sufficient (as an economic deterrent) vs most enemies. However, this is a simplistic model and doesn’t account for targeted attacks. It’s a complex problem space and has a lot of room to mature. I expect great changes in this space over the coming years.

Kim_Bruning 6y ago

Wrt uneducated vendors: Industrial Control systems tend to be built by people with an electrical background rather than an IT background, and they have their own culture, and strong Not Invented Here effect. A strong cross-disciplinary startup could make a killing in industrial automation. (And extant companies that remotely meet that criterion already seem to do so.)