7 ms·
I work on about 30 different power plants for which I have written software in the control systems. I can access all of them remotely, no way i’m flying/Driving
by rhodozelia 6y ago
I work on about 30 different power plants for which I have written software in the control systems. I can access all of them remotely, no way i’m flying/Driving to middle of nowhere to make a change, and no way those customers are going to pay me for three days what otherwise takes a few hours.
- usea 6y agoI worked on software that interfaced with industrial control systems on some large machinery that could definitely kill people (and sometimes did, anyway). We absolutely flew out to every location to make changes, around the world. Yes, it was a great cost of time and money. But it's just irresponsible to allow connections from the outside when something going wrong would kill people and cause a lot of damage.
- rhodozelia 6y agoOther than syncing a generator totally out of phase I can’t think of anything that could potentially hurt someone at the power plants I work on. If the machines are killing people sometimes that is probably a whole other level of risk and I agree not something I would want to touch remotely.
- AnthonyMouse 6y ago> Other than syncing a generator totally out of phase I can’t think of anything that could potentially hurt someone at the power plants I work on. Don't power plants have all kinds of potentially dangerous mechanical systems under computer control? Overpressure on a fuel line causing the pipe to burst and fill the building with natural gas, things like that?
- kortilla 6y agoThere aren’t nearly as many things as you might think that have unbounded failure modes like that. At the mechanical and electrical engineering level you have fuses, breakers, pressure relief valves, etc that make all of these failure scenarios just “the machine shut down”.
- TeMPOraL 6y agoAnd then the machine shuts down and you have to fly there anyway to fix it. This sounds like turning "an attacker may destroy power generation infrastructure and maybe kill people" problem into "an attacker may repeatedly DoS the power generation systems, likely damage it, and still probably cause loss of life by second-order effects". Doesn't feel all that different. To use an analogy: it's like saying that attackers accessing the company servers remotely aren't a problem, because the best they can do is to keep them in a reboot loop...
- daniellarusso 6y agoYour analogy made me chuckle.
- rhodozelia 6y agothe western grid has 258,000 MW of generation. renewables go up and down like yo-yos, power plants go on-line and offline all the time and a large and always increasing fraction of it is unscheduled and uncontrolled. Of course it depends on the size and location of the power plant, but there are many more small plants that can go offline without causing any significant effect to the grid than there are large ones, and the large ones probably have the resources to be manned 24/7 either on site or from a remote site over a dedicated and thereby secure fibre link. One take-away from this thread is that not all infrastructure is 'critical'.
- kortilla 6y agoCrappy analogy. There is a huge difference between being able to shut down some power generation that is a drop in the bucket on a grid vs overloading some equipment and killing people. One is life and death, another is lost revenue for a power plant.
- yodelshady 6y agoIf you think infosec people are paranoid, boy have you not met a good mechanical engineer yet. Down in the analogue world, random overpressures, voltage spikes, simple mechanical fatigure, etc happen all the time, so you're going to have to deal. I've written reports on why, regardless of what a sensor said, a simple static beam made of nonflammable materials neither a) spontaneously combusted, b) broke Clausius's principle and moved heat from a cold body to a hotter body. But we still checked.
- ezconnect 6y agoI agree on this, if something bad happened most critical system are designed to break somewhere to limit the catastrophe, or other times they are over engineered 1,000x because sometimes you can never tell what nature can do to your system.
- TeMPOraL 6y agoI wish to believe that. I'd expect it of good engineers. However, I don't trust systems to remain as robust as initially designed, after decades of undergoing constant pressure to reduce costs, make things more (fiscally) efficient, "cut out fat", etc.
- Arrath 6y agoEven the loss of institutional knowledge can degrade the safety of a system over time. Ever had to start maintaining a codebase after the greybeard who made it retired? Did you have any documentation to go off of? Anyone remaining in the company regard it as anything other than a production critical black box? Now imagine that codebase is an industrial process, with 20 years of growth, add-ons, changes. Probably some mismatching control systems, and god only knows if the as-builts or red line plans for the additions are all correct or rectified with each other..
- Arrath 6y agoCertainly the systems are designed to. But over the operational life, things happen. Systems degrade, maybe maintenance lapses a bit. Maybe, for example, a mechanic tired of trouble shooting an issue replaces the 9th blown burst disc in a row with one he just cut from the mud flap on his shop truck. NBD, the system was only running 5% over pressure anyway and he wants to go home. Later, this overpressure state backs up through the system and overheats a pump which, considering that it is pumping liquid explosives, isn't ideal. Maybe this trips a temperature sensor and the jury rigged burst disc is discovered. Maybe it doesn't, or the sensor was bypassed or silenced, and the pump explodes. I'm rambling, but this is a real world example. My point is that these various factors: designed safety monitors like sensors or fail-safes play hand in hand with over-engineered systems as well as the security of the control systems. Any one part of the system should not be allowed to lapse or be less secure with the idea that another level of the system will catch the problem before it becomes life endangering.
- HenryBemis 6y agoMechanical, chemical, regulating temperature, regulating flow, regulating speed. Imagine chemical X reached 10C over the recommended and starts a chain reaction. Or imagine sensors going offline. I was once working on a food-producing factory, and we had our shares of accidents, though not lethal. Any type of machinery with fluctuating power supply may be dangerous unless there are automated killswitches, release valves, etc. in place.
- Tepix 6y agoPower failures (especially long ones) can kill people
- roel_v 6y agoIf the power plants would shut down or be severely damaged, wouldn't the resultant lack of power cause all sorts of damage, including things like accident victims not being able to reach emergency services and all the other classic ripple failures? I can imagine there wouldn't be any direct casualties in the plant, but I can't imagine power plants that upon unplanned shutdown would not cause severe damage, including physical harm and death (albeit indirectly). Even power disruptions of a few hours cause deaths.
- yodelshady 6y agoSyncing a generator out of phase is exactly what was trialled in the Aurora Generator Test: https://en.wikipedia.org/wiki/Aurora_Generator_Test https://en.wikipedia.org/wiki/Aurora_Generator_Test Didn't go well for the jenny. (Though a turbine might have fared better)
- i_am_proteus 6y agoDoes your power plant deal with any steam? Gag a few safety valves and find out the hard way.
- adrianN 6y agoYour customers probably would be more willing to pay if they had to be insured against damages caused by hacking.
- rhodozelia 6y agoIt’s an interesting question i was contemplating while writing the post, is it possible to cause damage by hacking the plants I work on? Exceeding conditions set forth in Environmental certificates, yes, destroying the machinery ... would be possible only because some modern equipment is used and connected to the network, if the Ethernet ports on the devices containing the sync-checks were disconnected, or an older dumb device where any setting adjustments are made with a screwdriver, then it wouldn’t be possible to damage about half of the plants. The other half with oil pumps you could cause some significant damage by telling it the oil pump is running when it is not and suppressing all the bearing temperature protection. I wonder what if any cyber security requirements are imposed by their insurance companies.
- wahern 6y ago> would be possible only because some modern equipment is used and connected to the network [...] containing the sync-checks "How 30 Lines of Code Blew Up a 27-Ton Generator: A secret experiment in 2007 proved that hackers could devastate power grid equipment beyond repair—with a file no bigger than a gif": https://www.wired.com/story/how-30-lines-of-code-blew-up-27-ton-generator/ https://www.wired.com/story/how-30-lines-of-code-blew-up-27-... YouTube video of demonstration: https://youtu.be/LM8kLaJ2NDU https://youtu.be/LM8kLaJ2NDU
- mtsr 6y agoBut you’re forgetting second order damage. What if somebody takes out a large enough fraction of power plants in the middle of winter? Or in the middle of a heat wave in Texas? Casualties could be rather significant if the plants can’t be restored in time.
- AnthonyMouse 6y ago> Your customers probably would be more willing to pay if they had to be insured against damages caused by hacking. This is actually the other way around. Insurance is a moral hazard. If you're paying for the insurance anyway then taking steps to reduce the insurance company's risk rather than your own is throwing money down the drain. Meanwhile the insurance company as a third party has less control over your actions than your own company and is less able to make efficient trade offs, so any requirements they impose tend to be either ineffective or unnecessarily burdensome or both. The problem is that publicly traded or municipal utility companies have pretty much the same problem where the employees/managers don't pay personally for damage and the people who do are too diffuse or far away to understand their exposure well enough to mitigate it.
- jacobwilliamroy 6y agoI will take the free government money thank you very much. Maybe even hire another person to help make these physical site visits. If we don't take the government's money, they'll just spend it on weapons.
- rhodozelia 6y agoPrivate and public companies own power plants too.
- jacobwilliamroy 6y agoCool. Let's lobby for regulations that force those companies to hire us. We're gonna make so much money.
- rhodozelia 6y agoYou’d make more money as a plumber
- kenneth 6y agoYou're essentially you're willing to trade security of critical infrastructure (and potentially risk lives) for the convenience of being able to do your job from home.
- rhodozelia 6y agoThe plant owners have made that call. Is a vpn secure? If Tesla can update the software in a fleet of vehicles remotely why can’t I update the software in a power plant?
- waste_monk 6y ago>The plant owners have made that call. Arguably it should not be up to the owner entity as to the security requirements they need to meet. At least down here in Australia we're in the process of legislative reforms that significantly expand both the scope of critical infrastructure (most of which is privately owned) and the responsibilities and requirements of critical infrastructure operators (in terms of risk management, security plans, mandatory reporting, etc.) >Is a vpn secure? Teamviewer (which I understand was the vector used in the water system breach) or a VPN that hasn't passed the extensive testing required of secure cryptographic devices is not acceptable. IF the industrial control systems were treated as a classified information system, were linked using an approved and properly managed [1] secure cryptographic device (on both sides e.g. back to a secure control centre, not to your house so you can work from bed), and generally had the rest of the protective elements (e.g. requiring personnel working on it to hold a SECRET or equivalent clearance proportionate with the value of the system) in place, that might be OK. But otherwise my position remains the same - it should not be remotely controllable via the internet. >If Tesla can update the software in a fleet of vehicles remotely why can’t I update the software in a power plant? That might brick some cars on-board computers or "only" kill perhaps a few dozen people before people caught on and it was recalled - still tragic, but small in the grand scheme of things. Critical infrastructure is called that because if it goes wrong it can harm or kill people on a vast scale, so it should be held to far higher standards. [1] e.g. keymat issued by whatever your equivalent to the NSA is e.g. our Australian Signals Directorate, and otherwise verified to be secure.
- hathawsh 6y agoThanks for speaking up, rhodozelia. The currently prevailing voice of HN appears not to be on your side, but you have 30 customers who are on your side, and you probably represent the views of many. We need to hear more of your side.
- rhodozelia 6y agoI wish more attention (money) was given to cyber security on infrastructure projects, and all projects. Cyber security is a mess everywhere. How Many organizations in the USA have not been hacked or are Unhackable? The only one I would guess is google and maybe amazon? Banks and financial markets seem competent as well.
- SXX 6y agoBanks generally have terrible track of record for security across the globe. But they usually have large and agressive legal departments so no sensible white hat ever gonna touch them with a ten-foot pole. At least they wont ever be allowed to publish results of security audit.
- eecc 6y agoSo did anyone of your deployments feature on one of the presentations of @viss - https://youtu.be/hMtu7vV_HmY https://youtu.be/hMtu7vV_HmY
- bryanrasmussen 6y agoso if the customers would pay you to drive to the middle of nowhere to make the change you still wouldn't do it?
- zhte415 6y agoI'm a bit more concerned why one person is critical for 30 power stations. Surely there should be at least a few others that could be called. What if you were on holiday, incapacitated or just quit?
- Communitivity 6y agoSCADA systems can kill people, any computerized system that can kill people must be protected as a weapons platform, therefore SCADA system must be protected as if they were weapons platforms. Wait, isn't that a bit over-dramatic? I mean sure, that's true of nuclear systems. But I mean, my home thermostat? My car's smart key system? As for the thermostat - it's winter below freezing and a hacker turns your heat off during the night. Sure someone might wake up, and it would have to be very cold for you to freeze, but if you have someone old or a child then it could be life threatening. Depending on the heater it might be able to make it cause a fire. Many car's smart key system is on the same network in the car as everything else, including the brakes. Read more in https://money.cnn.com/2014/08/01/technology/security/most-hackable-cars/index.html https://money.cnn.com/2014/08/01/technology/security/most-ha... So no, it's not being overly dramatic to categorize any SCADA system as a weapons platform. It might be, as they say, 'an abundance of caution', but I am fine with erring on that side than erring the other way. Three Mile Island can be viewed as caused because one gauge was not being tracked correctly (where it was supposed to be was being tracked, not where it actually was). What if remote access was enabled on systems in a nuclear power plant, so that people didn't have to drive all the way there? A hacker might manage to replicate some version of Three Mile Island. I can hear the 'but it's secured' response from some already, but any system that can be accessed can be hacked, given enough time and resources. The key thing people should realize is that cybersecurity does not prevent hacking, it attempts to delay it until a real person can take corrective action to protect the system. This is why you have defense in depth and rings of security. Essentially, it is a door lock. The better the lock, the more skilled class of thief it will dissuade by being 'too hard to bother'. A skilled thief who has state of the art lock picks, or is willing to use a door ram, is not going to be deterred by even the best of locks (think organized crime or state actor hackers here). But if our imaginary door only opens one way no matter what, to let people out, and can physically never let people in - then even the greatest thief is SOL. That is the purpose of a data diode. The other other concern, and it's not minor, is that any time you start measuring safety in terms of cost effectiveness you are putting a value on human life. It is saying that is is cheaper to pay the settlements and/or fines resulting from deaths due to insufficient safety precautions than it is to make it safe to begin with.