4 ms·

I work in control systems OT space. A lot of distributed control systems and scada systems interface with the business layer in some fashion to provide access

by procarch2019 5y ago

I work in control systems OT space. A lot of distributed control systems and scada systems interface with the business layer in some fashion to provide access to time series and event data and to allow for alerts via email/mobile. Some people do this properly with good network segmentation, firewalls, A/V and patching, etc (there are several standards that dictate best practice). That said, even when doing it properly you're introducing attack vectors. I don't think it would be a firmware vulnerability, but instead something malicious affecting the computers they use to control the process.

motohagiography 5y ago
The reason I'm going for firmware is while the HMIs could have had a solarwinds style exposure, but that's just any generically wormable OS vulnerability, and not something that should cause a physical shutdown. To shutdown a pipeline, it's not a management console issue, hence why I'd speculate it's in the ICS devices themselves, which probably use uClinux toolchains on SoCs from one or two large vendors. I did some smart meter and ICS security work in the 00's, and there were a few vendors who would be strategic targets. The attack tools available now are unbelievably better, while the attack surface is pretty much the same due to the long lifecycles of ICS components, and considering today we've got cheap SDRs and gnuradio blocks for most wireless protocols, AVR tools, buspirate and the good/greatfet, ghidra/ida, and python for reverse engineering, the vulnerability research on this stuff moves way faster than the industry ability to respond. If this is a serious attack, the only way to respond will be if they are very lucky, it's a worm and they can stand up a honeynet with spare gear to catch a sample and any good infosec firm can pull it apart. But if it's an active APT group, there's probably a political solution, as given what's possible, this would seem to be just a shot over the bow.
- procarch2019 5y ago
  I get what you're saying and that could very well be the case, but I think the 'pipeline' as a whole requires a lot of handshaking between the different stations. They would not be able to do this without their supervisory control later (or at least it would be particularly difficult). That alone could have caused them to shut it down. Additionally, if there was a whiff of malicious software or unintended access I would imagine they would want to make sure it didn't get into other systems. That would involve isolating and possibly shutting down machines and equipment. I guess we'll see when they release more information. I would imagine that we'll get more details since this is critical infrastructure.
- rhodozelia 5y ago
  If the management console has a button or controls that would allow the person sitting at the management console to shut down the pipeline, which systems usually do have an emergency stop button in case there is an accident, then all you need is access to the management console to write one bit to the controller that says “operator pressed estop” No need for firmware vulnerabilities in VxWorks when there are internet connected windows pcs.
- RhodoGSA 5y ago
  Very interesting, kinda spooky. Peer-to-peer threats from a world power perspective seem to be less bullets and more code. Any cyber warfare would just end in both parties destroying critical infrastructure until there's none left. War of attrition, skipping completely past the military and affecting the civilian population directly.
sandworm101 5y ago
>> but instead something malicious affecting the computers they use to control the process. I bet there is a layer of windows XP machines involved in a legacy control system. XP machines that weren't supposed to connect to the internet somehow have malware on them. It doesn't even have to do anything. Simply the detection of anything in such circumstances is enough to warrant them being shut down.
- procarch2019 5y ago
  Totally agree, see it all the time. I even know of a few NT systems floating around out there. At least most companies are getting their IT involved to mitigate (usually they work with the vendor because they know nothing about control systems). They usually provide funding to the automation groups. People are starting to take it seriously.
bilbo0s 5y ago
This. I've said it a thousand times, all the security in the world will not defend a SCADA system if someone left TeamViewer running somewhere. Don't mean to pick on TeamViewer. It could be any number of packages, but I think security minded people get an idea of the type of attack vectors I'm talking about.
- procarch2019 5y ago
  It is mind boggling the lack of basic security principles some people have. I won't just put that on the plants and their IT/OT, or lack thereof. I've seen plenty of vendors and integrators do some cringe worthy stuff too.
  - rhodozelia 5y ago
    The whole automation industry is a security disaster but it is because security isn’t part of the deliverables for any party. It isn’t in the specs, civil, mechanical, electrical engineers it isn’t their responsibility. If the owner has an IT department they usually don’t want to be responsible for it either since locking things down leads to weird issues with legacy proprietary SCADA systems. There is no out of the box secure solution available yet. Rockwell certainly makes an attempt with their factory talk directory but I highly doubt that isn’t easily worked around somehow.
    - procarch2019 5y ago
      Yea, that is correct. I typically put together the solutions for new systems, including security. I give the sales team part numbers and hours for security software and related hardware. They then add that as an option to quotes. No principal automation engineer wants to take that on and no IT want to be involved. Also, when money is tight that’s an easy target for them to pass on. Luckily I’ve pushed enough over the years that we at least include A/V software as mandatory. I’ve been able to carve out a nice space within my company bridging the IT/OT divide. It’s been particularly good recently since the bigger companies are dictating good cyber practices, but rely on integrators and vendors to implement. I don’t think there will ever be an out of the box solution unless a system stands on its own, which is becoming increasingly harder with modernization and reliability efforts. Add on top of that privileged access, remote monitoring and support, automated (kind of) patching, etc. you have to interface with the IT side a bit.
tw04 5y ago
Why wouldn't you use a unidirectional connection for time series and event data? I understand why you might want to send things out to the rest of the world, I can't fathom why you wouldn't require physical access to have write access.
- exikyut 5y ago
  Genuine question (that I've been seriously wondering about for a long time): how do you implement validated attestation that a piece of log data has reached nonvolatile storage, triggered appropriate alarms, and that those alarm events have been acknowledged, while using a data diode type setup?
  - jtchang 5y ago
    Depends on your setup but a message bus architecture with polling would work.
  - 8note 5y ago
    What do you do when this attestation fails? Eg. A fox chewed through the cable and the ack can't be received.
  - mikewarot 5y ago
    If it is critical to have the log, it has to be local. Infrastructure shouldn't die if an internet connection goes down. You can sent the status of the log out through the data diode, along with a copy of the data.
- procarch2019 5y ago
  Some time series data interfaces only work with tcp comms, which means you can’t always rely on unidirectional networks. I agree you should use them where possible though. I replied to a comment on a dupe post regarding PAT, in which analysis is done on process data and fed back into the process to increase efficiency or yield. Obviously there are varying levels of criticality where the risk vs the business reward might not be worth it though.

motohagiography 5y ago

The reason I'm going for firmware is while the HMIs could have had a solarwinds style exposure, but that's just any generically wormable OS vulnerability, and not something that should cause a physical shutdown. To shutdown a pipeline, it's not a management console issue, hence why I'd speculate it's in the ICS devices themselves, which probably use uClinux toolchains on SoCs from one or two large vendors. I did some smart meter and ICS security work in the 00's, and there were a few vendors who would be strategic targets. The attack tools available now are unbelievably better, while the attack surface is pretty much the same due to the long lifecycles of ICS components, and considering today we've got cheap SDRs and gnuradio blocks for most wireless protocols, AVR tools, buspirate and the good/greatfet, ghidra/ida, and python for reverse engineering, the vulnerability research on this stuff moves way faster than the industry ability to respond. If this is a serious attack, the only way to respond will be if they are very lucky, it's a worm and they can stand up a honeynet with spare gear to catch a sample and any good infosec firm can pull it apart. But if it's an active APT group, there's probably a political solution, as given what's possible, this would seem to be just a shot over the bow.

procarch2019 5y ago
I get what you're saying and that could very well be the case, but I think the 'pipeline' as a whole requires a lot of handshaking between the different stations. They would not be able to do this without their supervisory control later (or at least it would be particularly difficult). That alone could have caused them to shut it down. Additionally, if there was a whiff of malicious software or unintended access I would imagine they would want to make sure it didn't get into other systems. That would involve isolating and possibly shutting down machines and equipment. I guess we'll see when they release more information. I would imagine that we'll get more details since this is critical infrastructure.
rhodozelia 5y ago
If the management console has a button or controls that would allow the person sitting at the management console to shut down the pipeline, which systems usually do have an emergency stop button in case there is an accident, then all you need is access to the management console to write one bit to the controller that says “operator pressed estop” No need for firmware vulnerabilities in VxWorks when there are internet connected windows pcs.
RhodoGSA 5y ago
Very interesting, kinda spooky. Peer-to-peer threats from a world power perspective seem to be less bullets and more code. Any cyber warfare would just end in both parties destroying critical infrastructure until there's none left. War of attrition, skipping completely past the military and affecting the civilian population directly.

sandworm101 5y ago

>> but instead something malicious affecting the computers they use to control the process. I bet there is a layer of windows XP machines involved in a legacy control system. XP machines that weren't supposed to connect to the internet somehow have malware on them. It doesn't even have to do anything. Simply the detection of anything in such circumstances is enough to warrant them being shut down.

procarch2019 5y ago
Totally agree, see it all the time. I even know of a few NT systems floating around out there. At least most companies are getting their IT involved to mitigate (usually they work with the vendor because they know nothing about control systems). They usually provide funding to the automation groups. People are starting to take it seriously.

bilbo0s 5y ago

This. I've said it a thousand times, all the security in the world will not defend a SCADA system if someone left TeamViewer running somewhere. Don't mean to pick on TeamViewer. It could be any number of packages, but I think security minded people get an idea of the type of attack vectors I'm talking about.

procarch2019 5y ago
It is mind boggling the lack of basic security principles some people have. I won't just put that on the plants and their IT/OT, or lack thereof. I've seen plenty of vendors and integrators do some cringe worthy stuff too.
- rhodozelia 5y ago
  The whole automation industry is a security disaster but it is because security isn’t part of the deliverables for any party. It isn’t in the specs, civil, mechanical, electrical engineers it isn’t their responsibility. If the owner has an IT department they usually don’t want to be responsible for it either since locking things down leads to weird issues with legacy proprietary SCADA systems. There is no out of the box secure solution available yet. Rockwell certainly makes an attempt with their factory talk directory but I highly doubt that isn’t easily worked around somehow.
  - procarch2019 5y ago
    Yea, that is correct. I typically put together the solutions for new systems, including security. I give the sales team part numbers and hours for security software and related hardware. They then add that as an option to quotes. No principal automation engineer wants to take that on and no IT want to be involved. Also, when money is tight that’s an easy target for them to pass on. Luckily I’ve pushed enough over the years that we at least include A/V software as mandatory. I’ve been able to carve out a nice space within my company bridging the IT/OT divide. It’s been particularly good recently since the bigger companies are dictating good cyber practices, but rely on integrators and vendors to implement. I don’t think there will ever be an out of the box solution unless a system stands on its own, which is becoming increasingly harder with modernization and reliability efforts. Add on top of that privileged access, remote monitoring and support, automated (kind of) patching, etc. you have to interface with the IT side a bit.

tw04 5y ago

Why wouldn't you use a unidirectional connection for time series and event data? I understand why you might want to send things out to the rest of the world, I can't fathom why you wouldn't require physical access to have write access.

exikyut 5y ago
Genuine question (that I've been seriously wondering about for a long time): how do you implement validated attestation that a piece of log data has reached nonvolatile storage, triggered appropriate alarms, and that those alarm events have been acknowledged, while using a data diode type setup?
- jtchang 5y ago
  Depends on your setup but a message bus architecture with polling would work.
- 8note 5y ago
  What do you do when this attestation fails? Eg. A fox chewed through the cable and the ack can't be received.
- mikewarot 5y ago
  If it is critical to have the log, it has to be local. Infrastructure shouldn't die if an internet connection goes down. You can sent the status of the log out through the data diode, along with a copy of the data.
procarch2019 5y ago
Some time series data interfaces only work with tcp comms, which means you can’t always rely on unidirectional networks. I agree you should use them where possible though. I replied to a comment on a dupe post regarding PAT, in which analysis is done on process data and fed back into the process to increase efficiency or yield. Obviously there are varying levels of criticality where the risk vs the business reward might not be worth it though.