8 ms·

Colonial Pipeline precisely does keep it's control network disconnected from the internet - the only thing that was ransomwared is their corporate network. They

Colonial Pipeline precisely does keep it's control network disconnected from the internet - the only thing that was ransomwared is their corporate network. They shut the pipelines down voluntarily to prevent further spread.

engineer_22 5y ago
Please explain why shutting down the pipeline will contain the hack?
- chiph 5y ago
  You need the SCADA systems to run the pipeline. They control the pumps, valves, product sequencing, etc. So Colonial purposely shut down the pipeline to prevent the SCADA system from getting affected, which might cause physical damage that truly would be a catastrophe.
  - deleted 5y ago
    [deleted]
  - refulgentis 5y ago
    I'm really confused: the pipeline is resilient to a hack: they just shut down the pipeline so it won't be 'affected' (hacked?)?
    - rckoepke 5y ago
      It can be already hacked but while power to the valves and pumps are removed then the SCADA system hacks can't cause physical damage. I haven't seen any evidence that the "OT" side of their network was compromised in a way that would cause physical damage, a la Stuxnet.
    - chiph 5y ago
      It was intended to be airgapped, but we're talking about a pipeline that is several thousand miles long, with many pumping stations and delivery terminals. All it would take is one of the SCADA systems at one of those locations to suddenly open a valve and dump petroleum out into the environment to cause a disaster. Or worse - rapidly open & close valves in rhythm, and the water hammer effect (the inertia of the petroleum in the pipeline) would cause the pipeline to destroy itself. The repair costs would be astronomical - you'd naturally have to repair the damaged sections, but then also re-test all the welds to see if any had been weakened by the pressure pulses.
      - rckoepke 5y ago
        It was not intended to be air-gapped. These systems generally communicate to business layers through firewalls. Onion-layer security rather than air gaps. Communication through the firewall isn't supposed to allow control over the valves, but it does communicate both ways (TCP/IP). This is the general practice in petrochemicals, at any rate.
      - WalterBright 5y ago
        In airplanes, this is dealt with by having an independent system monitoring things. Think what would happen if the autopilot suddenly went berserk and did a hardover.
  - avs733 5y ago
    And to add it’s perfectly possible that the pipeline networks were air gapped (Ed: which don’t believe them) but you still need to shut down. I could imagine a situation where information another network (e.g. orders or incoming flows from another customer or user) is necessary to run the pipeline but unavailable to use to operate the pipeline control system.
mulmen 5y ago
If we define critical system as "necessary to the operation of the business" then the corporate system is absolutely critical. It doesn't matter if the SCADA system is airgapped if you can shut down the capability by crashing the corporate systems.
- temp667 5y ago
  Their approach makes a lot of sense. Corp network hacked - so to be careful shut down pipeline until you've really made sure that you are fully safe pipeline side as there now may be more attack vectors.
croes 5y ago
Any source for that?
vsareto 5y ago
I saw Robert Lee of Dragos giving an opinion on that here https://www.reuters.com/business/energy/knowns-unknowns-about-hack-colonial-pipeline-2021-05-09/ https://www.reuters.com/business/energy/knowns-unknowns-abou... but did they confirm that officially? >Colonial has not given any public indication as to the reach of the ransomware outbreak, but Robert M. Lee, chief executive of cybersecurity firm Dragos, said he believed Colonial's operations network was shut down proactively "to make sure that nothing spread into those systems."
radicaldreamer 5y ago
I have no idea why they would do that unless the system was not airgapped properly or it was hard to untangle the admin network from the control network (in which case, the control network is effectively not airgapped).
- avs733 5y ago
  Flash drives. I used to work in fabs and every couple of years some tool or other would get a virus, sometimes it spread through the network.
  - rckoepke 5y ago
    USB ports are generally disabled in BIOS or purposefully physically damaged on most OT systems I've worked on for oil/gas/chemicals. Many places are fond of using epoxy to block the ports.
    - avs733 5y ago
      I like those people. The problem being sometimes you need logs or data off tools. I’m far from an IT wizard so I don’t know what other solutions exist but the flash drives to get stuff off tools was the easiest
      - rckoepke 5y ago
        It makes some things more difficult. CD/DVD's are generally used instead. Sometimes other computers could be connected but in that case there would be some organizational procedure for attempting to make sure that other computer was as low risk as possible. You can't eliminate the possibility of malicious action, Stuxnet proves that. It's my opinion that at least for critical infrastructure we can probably make things much more difficult for our adversaries at a relatively low cost. This pipeline is purported to carry half the gasoline/diesel/heating oil to the east coast, but I'd be lying if I said I knew exactly where the cost-benefit equilibrium should land.
rckoepke 5y ago
I built some of the SCADA and IT systems for Colonial Pipeline. Many industrial SCADA systems (nearly all) send data from their "OT" systems (PLC/DCS/SCADA) to their "IT" and business layers (Historians/Timeseries Databases, Dashboards, Power BI/etc). This almost always happens through a two-way link (think TCP/IP, HTTP). While the software should not allow data flow backwards, the hardware absolutely does. So how much do you trust the software? I often advocate that industrial SCADA systems utilize "data-diodes", one-way opto-isolators, or other physically verifiable methods of confirming that no information/data/instructions can get from a "higher" layer (OSI Pi, PowerBI) to a lower layer (Allen-Bradley PLC, Siemens PLC, Emerson DeltaV DCS, etc). Convincing the powers-that-be to do this has been incredibly impossible in most places and a large reason why I'm trying to transition to a different space - I simply have had ethical concerns about providing engineering services to critical infrastructure without building in best practices. Stuxnet was over a decade ago - I don't understand how these protections aren't mandated by the DHS already. Disclaimer: I don't think it's reasonable for non-involved people to assume the OT side has been compromised. I do think Colonial will need some time to verify the integrity of their SCADA systems and it makes sense to keep the power to the physical devices (valves/pumps) offline until they do. I understand why they chose to shut down but I don't think there's any evidence that they'll be unable to start back up again. Lastly, I saw a quote in one article: >>> Digital Shadows thinks the Colonial Pipeline cyber-attack has come about due to the coronavirus pandemic - the rise of engineers remotely accessing control systems for the pipeline from home. I strongly doubt this. It's possible, of course. But it's extremely unlikely to me that employees would have remotely accessed OT/SCADA systems from home. No one I've worked with has had that capability enabled. Many companies use products which have been shown to have flaws, like Citrix or various corporate VPNs. These could be compromised to get access "closer" to the OT layers but never directly into it. Onion layer security is very much practiced everywhere I've been. Edit: I have heard of some petrochemical facilities moving towards allowing operators and engineers to manipulate valves/pumps on their iPhones. This horrifies me for many reasons. I've never actually seen it implemented and I always bring up Stuxnet when I hear people mention it. I personally believe that DHS should make this sort of thing illegal for critical infrastructure. Many good engineers disagree with me.
- sebmellen 5y ago
SvenMarquardt 5y ago
Why is the network constructed in such a way that it allows things to spread?
- tru3_power 5y ago
  Super common. A lot of companies have this hard on the outside gooey on the inside model (aka flat network structure).
  - rckoepke 5y ago
    Most petrochemical companies have an onion structure. There are lots of layers of firewalls with what are supposed to be limited communication paths for specific applications. We need to move to using physical layers where data can only be transmitted in one direction (and then use something like UDP)
riffic 5y ago
it's always means it is (or it was); it's never possessive.
dang 5y ago
We detached this subthread from https://news.ycombinator.com/item?id=27101293 https://news.ycombinator.com/item?id=27101293, which was a generic subthread, because the discussion turned more relevant at this point. (To a first approximation, generic == less interesting and specific == more interesting on HN: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=generic%20discussion%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...)

engineer_22 5y ago

Please explain why shutting down the pipeline will contain the hack?

chiph 5y ago
You need the SCADA systems to run the pipeline. They control the pumps, valves, product sequencing, etc. So Colonial purposely shut down the pipeline to prevent the SCADA system from getting affected, which might cause physical damage that truly would be a catastrophe.
- deleted 5y ago
  [deleted]
- refulgentis 5y ago
  I'm really confused: the pipeline is resilient to a hack: they just shut down the pipeline so it won't be 'affected' (hacked?)?
  - rckoepke 5y ago
    It can be already hacked but while power to the valves and pumps are removed then the SCADA system hacks can't cause physical damage. I haven't seen any evidence that the "OT" side of their network was compromised in a way that would cause physical damage, a la Stuxnet.
  - chiph 5y ago
    It was intended to be airgapped, but we're talking about a pipeline that is several thousand miles long, with many pumping stations and delivery terminals. All it would take is one of the SCADA systems at one of those locations to suddenly open a valve and dump petroleum out into the environment to cause a disaster. Or worse - rapidly open & close valves in rhythm, and the water hammer effect (the inertia of the petroleum in the pipeline) would cause the pipeline to destroy itself. The repair costs would be astronomical - you'd naturally have to repair the damaged sections, but then also re-test all the welds to see if any had been weakened by the pressure pulses.
    - rckoepke 5y ago
      It was not intended to be air-gapped. These systems generally communicate to business layers through firewalls. Onion-layer security rather than air gaps. Communication through the firewall isn't supposed to allow control over the valves, but it does communicate both ways (TCP/IP). This is the general practice in petrochemicals, at any rate.
    - WalterBright 5y ago
      In airplanes, this is dealt with by having an independent system monitoring things. Think what would happen if the autopilot suddenly went berserk and did a hardover.
- avs733 5y ago
  And to add it’s perfectly possible that the pipeline networks were air gapped (Ed: which don’t believe them) but you still need to shut down. I could imagine a situation where information another network (e.g. orders or incoming flows from another customer or user) is necessary to run the pipeline but unavailable to use to operate the pipeline control system.

mulmen 5y ago

If we define critical system as "necessary to the operation of the business" then the corporate system is absolutely critical. It doesn't matter if the SCADA system is airgapped if you can shut down the capability by crashing the corporate systems.

temp667 5y ago
Their approach makes a lot of sense. Corp network hacked - so to be careful shut down pipeline until you've really made sure that you are fully safe pipeline side as there now may be more attack vectors.

croes 5y ago

Any source for that?

vsareto 5y ago

I saw Robert Lee of Dragos giving an opinion on that here https://www.reuters.com/business/energy/knowns-unknowns-about-hack-colonial-pipeline-2021-05-09/ https://www.reuters.com/business/energy/knowns-unknowns-abou... but did they confirm that officially? >Colonial has not given any public indication as to the reach of the ransomware outbreak, but Robert M. Lee, chief executive of cybersecurity firm Dragos, said he believed Colonial's operations network was shut down proactively "to make sure that nothing spread into those systems."

radicaldreamer 5y ago

I have no idea why they would do that unless the system was not airgapped properly or it was hard to untangle the admin network from the control network (in which case, the control network is effectively not airgapped).

avs733 5y ago
Flash drives. I used to work in fabs and every couple of years some tool or other would get a virus, sometimes it spread through the network.
- rckoepke 5y ago
  USB ports are generally disabled in BIOS or purposefully physically damaged on most OT systems I've worked on for oil/gas/chemicals. Many places are fond of using epoxy to block the ports.
  - avs733 5y ago
    I like those people. The problem being sometimes you need logs or data off tools. I’m far from an IT wizard so I don’t know what other solutions exist but the flash drives to get stuff off tools was the easiest
    - rckoepke 5y ago
      It makes some things more difficult. CD/DVD's are generally used instead. Sometimes other computers could be connected but in that case there would be some organizational procedure for attempting to make sure that other computer was as low risk as possible. You can't eliminate the possibility of malicious action, Stuxnet proves that. It's my opinion that at least for critical infrastructure we can probably make things much more difficult for our adversaries at a relatively low cost. This pipeline is purported to carry half the gasoline/diesel/heating oil to the east coast, but I'd be lying if I said I knew exactly where the cost-benefit equilibrium should land.

rckoepke 5y ago

I built some of the SCADA and IT systems for Colonial Pipeline. Many industrial SCADA systems (nearly all) send data from their "OT" systems (PLC/DCS/SCADA) to their "IT" and business layers (Historians/Timeseries Databases, Dashboards, Power BI/etc). This almost always happens through a two-way link (think TCP/IP, HTTP). While the software should not allow data flow backwards, the hardware absolutely does. So how much do you trust the software? I often advocate that industrial SCADA systems utilize "data-diodes", one-way opto-isolators, or other physically verifiable methods of confirming that no information/data/instructions can get from a "higher" layer (OSI Pi, PowerBI) to a lower layer (Allen-Bradley PLC, Siemens PLC, Emerson DeltaV DCS, etc). Convincing the powers-that-be to do this has been incredibly impossible in most places and a large reason why I'm trying to transition to a different space - I simply have had ethical concerns about providing engineering services to critical infrastructure without building in best practices. Stuxnet was over a decade ago - I don't understand how these protections aren't mandated by the DHS already. Disclaimer: I don't think it's reasonable for non-involved people to assume the OT side has been compromised. I do think Colonial will need some time to verify the integrity of their SCADA systems and it makes sense to keep the power to the physical devices (valves/pumps) offline until they do. I understand why they chose to shut down but I don't think there's any evidence that they'll be unable to start back up again. Lastly, I saw a quote in one article: >>> Digital Shadows thinks the Colonial Pipeline cyber-attack has come about due to the coronavirus pandemic - the rise of engineers remotely accessing control systems for the pipeline from home. I strongly doubt this. It's possible, of course. But it's extremely unlikely to me that employees would have remotely accessed OT/SCADA systems from home. No one I've worked with has had that capability enabled. Many companies use products which have been shown to have flaws, like Citrix or various corporate VPNs. These could be compromised to get access "closer" to the OT layers but never directly into it. Onion layer security is very much practiced everywhere I've been. Edit: I have heard of some petrochemical facilities moving towards allowing operators and engineers to manipulate valves/pumps on their iPhones. This horrifies me for many reasons. I've never actually seen it implemented and I always bring up Stuxnet when I hear people mention it. I personally believe that DHS should make this sort of thing illegal for critical infrastructure. Many good engineers disagree with me.

sebmellen 5y ago

SvenMarquardt 5y ago

Why is the network constructed in such a way that it allows things to spread?

tru3_power 5y ago
Super common. A lot of companies have this hard on the outside gooey on the inside model (aka flat network structure).
- rckoepke 5y ago
  Most petrochemical companies have an onion structure. There are lots of layers of firewalls with what are supposed to be limited communication paths for specific applications. We need to move to using physical layers where data can only be transmitted in one direction (and then use something like UDP)

riffic 5y ago

it's always means it is (or it was); it's never possessive.

dang 5y ago

We detached this subthread from https://news.ycombinator.com/item?id=27101293 https://news.ycombinator.com/item?id=27101293, which was a generic subthread, because the discussion turned more relevant at this point. (To a first approximation, generic == less interesting and specific == more interesting on HN: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=generic%20discussion%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...)