6 ms·

I built some of the SCADA and IT systems for Colonial Pipeline. Many industrial SCADA systems (nearly all) send data from their "OT" systems (PLC/DCS/SCADA) to

I built some of the SCADA and IT systems for Colonial Pipeline. Many industrial SCADA systems (nearly all) send data from their "OT" systems (PLC/DCS/SCADA) to their "IT" and business layers (Historians/Timeseries Databases, Dashboards, Power BI/etc). This almost always happens through a two-way link (think TCP/IP, HTTP). While the software should not allow data flow backwards, the hardware absolutely does. So how much do you trust the software? I often advocate that industrial SCADA systems utilize "data-diodes", one-way opto-isolators, or other physically verifiable methods of confirming that no information/data/instructions can get from a "higher" layer (OSI Pi, PowerBI) to a lower layer (Allen-Bradley PLC, Siemens PLC, Emerson DeltaV DCS, etc). Convincing the powers-that-be to do this has been incredibly impossible in most places and a large reason why I'm trying to transition to a different space - I simply have had ethical concerns about providing engineering services to critical infrastructure without building in best practices. Stuxnet was over a decade ago - I don't understand how these protections aren't mandated by the DHS already. Disclaimer: I don't think it's reasonable for non-involved people to assume the OT side has been compromised. I do think Colonial will need some time to verify the integrity of their SCADA systems and it makes sense to keep the power to the physical devices (valves/pumps) offline until they do. I understand why they chose to shut down but I don't think there's any evidence that they'll be unable to start back up again. Lastly, I saw a quote in one article: >>> Digital Shadows thinks the Colonial Pipeline cyber-attack has come about due to the coronavirus pandemic - the rise of engineers remotely accessing control systems for the pipeline from home. I strongly doubt this. It's possible, of course. But it's extremely unlikely to me that employees would have remotely accessed OT/SCADA systems from home. No one I've worked with has had that capability enabled. Many companies use products which have been shown to have flaws, like Citrix or various corporate VPNs. These could be compromised to get access "closer" to the OT layers but never directly into it. Onion layer security is very much practiced everywhere I've been. Edit: I have heard of some petrochemical facilities moving towards allowing operators and engineers to manipulate valves/pumps on their iPhones. This horrifies me for many reasons. I've never actually seen it implemented and I always bring up Stuxnet when I hear people mention it. I personally believe that DHS should make this sort of thing illegal for critical infrastructure. Many good engineers disagree with me.

sebmellen 5y ago
Do you think Colonial identified some "physical world" risk, as in the possibility of a pressure overload or pipeline leak? I imagine that verifying the integrity of these SCADA systems is a very complex task, so I'm wondering if they've already identified a possible attack vector/entry point or if this was entirely preventative.
- rckoepke 5y ago
  I have no idea. Shutting down preventatively would be smart, and they had good leadership in their IT space while I was there. Friendly people who could make the hard decisions quickly, weren't afraid to pick up the phones to call people, and supported the growth of struggling employees without letting shoddy work get approved. They were also good at managing large multi-year and nation-wide project campaigns - a rare skill in this world. That said, determining whether or not a system was compromised can be incredibly difficult. I'm sure they'll face massive pressure to turn the pipeline back on as it does supply almost half of the east coast with oil. I wouldn't want to be the person who has to make that call when it's impossible to prove a negative. CPC had two explosions a few years back which caused gasoline shortages in new england, that may provide indication of the scale of disruption to expect.
  - WalterBright 5y ago
    That's why physical write-enable switches are a must for ROMs. If it's "off", the malware won't survive a reboot.
  - sebmellen 5y ago
    Thanks for the response. It's amazing to have a community where "subject-matter experts" like yourself just pop up. I'm quite surprised and comforted to hear that the leadership there is competent and knows how to manage people. I've heard from friends/acquaintances who have worked in the energy industry about how terribly things are put together on an IT front (PG&E being a prime culprit), so I was expecting the same here. I really like your "data-diodes" concept. Interested to see if such a thing takes off especially as these attacks evolve.

sebmellen 5y ago

Do you think Colonial identified some "physical world" risk, as in the possibility of a pressure overload or pipeline leak? I imagine that verifying the integrity of these SCADA systems is a very complex task, so I'm wondering if they've already identified a possible attack vector/entry point or if this was entirely preventative.

rckoepke 5y ago
I have no idea. Shutting down preventatively would be smart, and they had good leadership in their IT space while I was there. Friendly people who could make the hard decisions quickly, weren't afraid to pick up the phones to call people, and supported the growth of struggling employees without letting shoddy work get approved. They were also good at managing large multi-year and nation-wide project campaigns - a rare skill in this world. That said, determining whether or not a system was compromised can be incredibly difficult. I'm sure they'll face massive pressure to turn the pipeline back on as it does supply almost half of the east coast with oil. I wouldn't want to be the person who has to make that call when it's impossible to prove a negative. CPC had two explosions a few years back which caused gasoline shortages in new england, that may provide indication of the scale of disruption to expect.
- WalterBright 5y ago
  That's why physical write-enable switches are a must for ROMs. If it's "off", the malware won't survive a reboot.
- sebmellen 5y ago
  Thanks for the response. It's amazing to have a community where "subject-matter experts" like yourself just pop up. I'm quite surprised and comforted to hear that the leadership there is competent and knows how to manage people. I've heard from friends/acquaintances who have worked in the energy industry about how terribly things are put together on an IT front (PG&E being a prime culprit), so I was expecting the same here. I really like your "data-diodes" concept. Interested to see if such a thing takes off especially as these attacks evolve.