5 ms·

I worked on software that interfaced with industrial control systems on some large machinery that could definitely kill people (and sometimes did, anyway). We a

by usea 6y ago

I worked on software that interfaced with industrial control systems on some large machinery that could definitely kill people (and sometimes did, anyway). We absolutely flew out to every location to make changes, around the world. Yes, it was a great cost of time and money. But it's just irresponsible to allow connections from the outside when something going wrong would kill people and cause a lot of damage.

rhodozelia 6y ago
Other than syncing a generator totally out of phase I can’t think of anything that could potentially hurt someone at the power plants I work on. If the machines are killing people sometimes that is probably a whole other level of risk and I agree not something I would want to touch remotely.
- AnthonyMouse 6y ago
  > Other than syncing a generator totally out of phase I can’t think of anything that could potentially hurt someone at the power plants I work on. Don't power plants have all kinds of potentially dangerous mechanical systems under computer control? Overpressure on a fuel line causing the pipe to burst and fill the building with natural gas, things like that?
  - kortilla 6y ago
    There aren’t nearly as many things as you might think that have unbounded failure modes like that. At the mechanical and electrical engineering level you have fuses, breakers, pressure relief valves, etc that make all of these failure scenarios just “the machine shut down”.
    - TeMPOraL 6y ago
      And then the machine shuts down and you have to fly there anyway to fix it. This sounds like turning "an attacker may destroy power generation infrastructure and maybe kill people" problem into "an attacker may repeatedly DoS the power generation systems, likely damage it, and still probably cause loss of life by second-order effects". Doesn't feel all that different. To use an analogy: it's like saying that attackers accessing the company servers remotely aren't a problem, because the best they can do is to keep them in a reboot loop...
      - daniellarusso 6y ago
        Your analogy made me chuckle.
      - rhodozelia 6y ago
        the western grid has 258,000 MW of generation. renewables go up and down like yo-yos, power plants go on-line and offline all the time and a large and always increasing fraction of it is unscheduled and uncontrolled. Of course it depends on the size and location of the power plant, but there are many more small plants that can go offline without causing any significant effect to the grid than there are large ones, and the large ones probably have the resources to be manned 24/7 either on site or from a remote site over a dedicated and thereby secure fibre link. One take-away from this thread is that not all infrastructure is 'critical'.
      - kortilla 6y ago
        Crappy analogy. There is a huge difference between being able to shut down some power generation that is a drop in the bucket on a grid vs overloading some equipment and killing people. One is life and death, another is lost revenue for a power plant.
  - yodelshady 6y ago
    If you think infosec people are paranoid, boy have you not met a good mechanical engineer yet. Down in the analogue world, random overpressures, voltage spikes, simple mechanical fatigure, etc happen all the time, so you're going to have to deal. I've written reports on why, regardless of what a sensor said, a simple static beam made of nonflammable materials neither a) spontaneously combusted, b) broke Clausius's principle and moved heat from a cold body to a hotter body. But we still checked.
    - ezconnect 6y ago
      I agree on this, if something bad happened most critical system are designed to break somewhere to limit the catastrophe, or other times they are over engineered 1,000x because sometimes you can never tell what nature can do to your system.
      - TeMPOraL 6y ago
        I wish to believe that. I'd expect it of good engineers. However, I don't trust systems to remain as robust as initially designed, after decades of undergoing constant pressure to reduce costs, make things more (fiscally) efficient, "cut out fat", etc.
        Arrath 6y ago
        Even the loss of institutional knowledge can degrade the safety of a system over time. Ever had to start maintaining a codebase after the greybeard who made it retired? Did you have any documentation to go off of? Anyone remaining in the company regard it as anything other than a production critical black box? Now imagine that codebase is an industrial process, with 20 years of growth, add-ons, changes. Probably some mismatching control systems, and god only knows if the as-builts or red line plans for the additions are all correct or rectified with each other..
      - Arrath 6y ago
        Certainly the systems are designed to. But over the operational life, things happen. Systems degrade, maybe maintenance lapses a bit. Maybe, for example, a mechanic tired of trouble shooting an issue replaces the 9th blown burst disc in a row with one he just cut from the mud flap on his shop truck. NBD, the system was only running 5% over pressure anyway and he wants to go home. Later, this overpressure state backs up through the system and overheats a pump which, considering that it is pumping liquid explosives, isn't ideal. Maybe this trips a temperature sensor and the jury rigged burst disc is discovered. Maybe it doesn't, or the sensor was bypassed or silenced, and the pump explodes. I'm rambling, but this is a real world example. My point is that these various factors: designed safety monitors like sensors or fail-safes play hand in hand with over-engineered systems as well as the security of the control systems. Any one part of the system should not be allowed to lapse or be less secure with the idea that another level of the system will catch the problem before it becomes life endangering.
  - HenryBemis 6y ago
    Mechanical, chemical, regulating temperature, regulating flow, regulating speed. Imagine chemical X reached 10C over the recommended and starts a chain reaction. Or imagine sensors going offline. I was once working on a food-producing factory, and we had our shares of accidents, though not lethal. Any type of machinery with fluctuating power supply may be dangerous unless there are automated killswitches, release valves, etc. in place.
- Tepix 6y ago
  Power failures (especially long ones) can kill people
- roel_v 6y ago
  If the power plants would shut down or be severely damaged, wouldn't the resultant lack of power cause all sorts of damage, including things like accident victims not being able to reach emergency services and all the other classic ripple failures? I can imagine there wouldn't be any direct casualties in the plant, but I can't imagine power plants that upon unplanned shutdown would not cause severe damage, including physical harm and death (albeit indirectly). Even power disruptions of a few hours cause deaths.
- yodelshady 6y ago
  Syncing a generator out of phase is exactly what was trialled in the Aurora Generator Test: https://en.wikipedia.org/wiki/Aurora_Generator_Test https://en.wikipedia.org/wiki/Aurora_Generator_Test Didn't go well for the jenny. (Though a turbine might have fared better)
- i_am_proteus 6y ago
  Does your power plant deal with any steam? Gag a few safety valves and find out the hard way.

by usea 6y ago

rhodozelia 6y ago

Other than syncing a generator totally out of phase I can’t think of anything that could potentially hurt someone at the power plants I work on. If the machines are killing people sometimes that is probably a whole other level of risk and I agree not something I would want to touch remotely.

AnthonyMouse 6y ago
> Other than syncing a generator totally out of phase I can’t think of anything that could potentially hurt someone at the power plants I work on. Don't power plants have all kinds of potentially dangerous mechanical systems under computer control? Overpressure on a fuel line causing the pipe to burst and fill the building with natural gas, things like that?
- kortilla 6y ago
  There aren’t nearly as many things as you might think that have unbounded failure modes like that. At the mechanical and electrical engineering level you have fuses, breakers, pressure relief valves, etc that make all of these failure scenarios just “the machine shut down”.
  - TeMPOraL 6y ago
    And then the machine shuts down and you have to fly there anyway to fix it. This sounds like turning "an attacker may destroy power generation infrastructure and maybe kill people" problem into "an attacker may repeatedly DoS the power generation systems, likely damage it, and still probably cause loss of life by second-order effects". Doesn't feel all that different. To use an analogy: it's like saying that attackers accessing the company servers remotely aren't a problem, because the best they can do is to keep them in a reboot loop...
    - daniellarusso 6y ago
      Your analogy made me chuckle.
    - rhodozelia 6y ago
      the western grid has 258,000 MW of generation. renewables go up and down like yo-yos, power plants go on-line and offline all the time and a large and always increasing fraction of it is unscheduled and uncontrolled. Of course it depends on the size and location of the power plant, but there are many more small plants that can go offline without causing any significant effect to the grid than there are large ones, and the large ones probably have the resources to be manned 24/7 either on site or from a remote site over a dedicated and thereby secure fibre link. One take-away from this thread is that not all infrastructure is 'critical'.
    - kortilla 6y ago
      Crappy analogy. There is a huge difference between being able to shut down some power generation that is a drop in the bucket on a grid vs overloading some equipment and killing people. One is life and death, another is lost revenue for a power plant.
- yodelshady 6y ago
  If you think infosec people are paranoid, boy have you not met a good mechanical engineer yet. Down in the analogue world, random overpressures, voltage spikes, simple mechanical fatigure, etc happen all the time, so you're going to have to deal. I've written reports on why, regardless of what a sensor said, a simple static beam made of nonflammable materials neither a) spontaneously combusted, b) broke Clausius's principle and moved heat from a cold body to a hotter body. But we still checked.
  - ezconnect 6y ago
    I agree on this, if something bad happened most critical system are designed to break somewhere to limit the catastrophe, or other times they are over engineered 1,000x because sometimes you can never tell what nature can do to your system.
    - TeMPOraL 6y ago
      I wish to believe that. I'd expect it of good engineers. However, I don't trust systems to remain as robust as initially designed, after decades of undergoing constant pressure to reduce costs, make things more (fiscally) efficient, "cut out fat", etc.
      - Arrath 6y ago
        Even the loss of institutional knowledge can degrade the safety of a system over time. Ever had to start maintaining a codebase after the greybeard who made it retired? Did you have any documentation to go off of? Anyone remaining in the company regard it as anything other than a production critical black box? Now imagine that codebase is an industrial process, with 20 years of growth, add-ons, changes. Probably some mismatching control systems, and god only knows if the as-builts or red line plans for the additions are all correct or rectified with each other..
    - Arrath 6y ago
      Certainly the systems are designed to. But over the operational life, things happen. Systems degrade, maybe maintenance lapses a bit. Maybe, for example, a mechanic tired of trouble shooting an issue replaces the 9th blown burst disc in a row with one he just cut from the mud flap on his shop truck. NBD, the system was only running 5% over pressure anyway and he wants to go home. Later, this overpressure state backs up through the system and overheats a pump which, considering that it is pumping liquid explosives, isn't ideal. Maybe this trips a temperature sensor and the jury rigged burst disc is discovered. Maybe it doesn't, or the sensor was bypassed or silenced, and the pump explodes. I'm rambling, but this is a real world example. My point is that these various factors: designed safety monitors like sensors or fail-safes play hand in hand with over-engineered systems as well as the security of the control systems. Any one part of the system should not be allowed to lapse or be less secure with the idea that another level of the system will catch the problem before it becomes life endangering.
- HenryBemis 6y ago
  Mechanical, chemical, regulating temperature, regulating flow, regulating speed. Imagine chemical X reached 10C over the recommended and starts a chain reaction. Or imagine sensors going offline. I was once working on a food-producing factory, and we had our shares of accidents, though not lethal. Any type of machinery with fluctuating power supply may be dangerous unless there are automated killswitches, release valves, etc. in place.
Tepix 6y ago
Power failures (especially long ones) can kill people
roel_v 6y ago
If the power plants would shut down or be severely damaged, wouldn't the resultant lack of power cause all sorts of damage, including things like accident victims not being able to reach emergency services and all the other classic ripple failures? I can imagine there wouldn't be any direct casualties in the plant, but I can't imagine power plants that upon unplanned shutdown would not cause severe damage, including physical harm and death (albeit indirectly). Even power disruptions of a few hours cause deaths.
yodelshady 6y ago
Syncing a generator out of phase is exactly what was trialled in the Aurora Generator Test: https://en.wikipedia.org/wiki/Aurora_Generator_Test https://en.wikipedia.org/wiki/Aurora_Generator_Test Didn't go well for the jenny. (Though a turbine might have fared better)
i_am_proteus 6y ago
Does your power plant deal with any steam? Gag a few safety valves and find out the hard way.