8 ms·
I've said it before and I'll say it again. The only way critical infrastructure should be connected to the internet is via data diode [1]. Otherwise it should b
by waste_monk 6y ago
I've said it before and I'll say it again. The only way critical infrastructure should be connected to the internet is via data diode [1]. Otherwise it should be air-gapped.
It still lets you remotely monitor systems, but if you want to adjust the operation of the critical infrastructure you have to physically go there and do it in person. People often complain that this is inefficient, but if your own workers are unable to remotely affect a system then neither can an attacker - given the consequences of an attacker compromising a critical infrastructure I think the inconvenience is a fair price to pay.
[1] for those unfamiliar, this is typically implemented using two computers connected via fibreoptic cable, such that there are only transmitters (lasers) on one side and only receivers (photodiodes) on the other side. Thus the information can physically only travel in one direction. The computers at each end handle conversion from connection-oriented protocols (e.g. TCP) to connectionless (e.g. UDP) while maintaining reliability.
- rhodozelia 6y agoI work on about 30 different power plants for which I have written software in the control systems. I can access all of them remotely, no way i’m flying/Driving to middle of nowhere to make a change, and no way those customers are going to pay me for three days what otherwise takes a few hours.
- usea 6y agoI worked on software that interfaced with industrial control systems on some large machinery that could definitely kill people (and sometimes did, anyway). We absolutely flew out to every location to make changes, around the world. Yes, it was a great cost of time and money. But it's just irresponsible to allow connections from the outside when something going wrong would kill people and cause a lot of damage.
- rhodozelia 6y agoOther than syncing a generator totally out of phase I can’t think of anything that could potentially hurt someone at the power plants I work on. If the machines are killing people sometimes that is probably a whole other level of risk and I agree not something I would want to touch remotely.
- AnthonyMouse 6y ago> Other than syncing a generator totally out of phase I can’t think of anything that could potentially hurt someone at the power plants I work on. Don't power plants have all kinds of potentially dangerous mechanical systems under computer control? Overpressure on a fuel line causing the pipe to burst and fill the building with natural gas, things like that?
- kortilla 6y agoThere aren’t nearly as many things as you might think that have unbounded failure modes like that. At the mechanical and electrical engineering level you have fuses, breakers, pressure relief valves, etc that make all of these failure scenarios just “the machine shut down”.
- TeMPOraL 6y agoAnd then the machine shuts down and you have to fly there anyway to fix it. This sounds like turning "an attacker may destroy power generation infrastructure and maybe kill people" problem into "an attacker may repeatedly DoS the power generation systems, likely damage it, and still probably cause loss of life by second-order effects". Doesn't feel all that different. To use an analogy: it's like saying that attackers accessing the company servers remotely aren't a problem, because the best they can do is to keep them in a reboot loop...
- daniellarusso 6y agoYour analogy made me chuckle.
- rhodozelia 6y agothe western grid has 258,000 MW of generation. renewables go up and down like yo-yos, power plants go on-line and offline all the time and a large and always increasing fraction of it is unscheduled and uncontrolled. Of course it depends on the size and location of the power plant, but there are many more small plants that can go offline without causing any significant effect to the grid than there are large ones, and the large ones probably have the resources to be manned 24/7 either on site or from a remote site over a dedicated and thereby secure fibre link. One take-away from this thread is that not all infrastructure is 'critical'.
- adrianN 6y agoYour customers probably would be more willing to pay if they had to be insured against damages caused by hacking.
- rhodozelia 6y agoIt’s an interesting question i was contemplating while writing the post, is it possible to cause damage by hacking the plants I work on? Exceeding conditions set forth in Environmental certificates, yes, destroying the machinery ... would be possible only because some modern equipment is used and connected to the network, if the Ethernet ports on the devices containing the sync-checks were disconnected, or an older dumb device where any setting adjustments are made with a screwdriver, then it wouldn’t be possible to damage about half of the plants. The other half with oil pumps you could cause some significant damage by telling it the oil pump is running when it is not and suppressing all the bearing temperature protection. I wonder what if any cyber security requirements are imposed by their insurance companies.
- wahern 6y ago> would be possible only because some modern equipment is used and connected to the network [...] containing the sync-checks "How 30 Lines of Code Blew Up a 27-Ton Generator: A secret experiment in 2007 proved that hackers could devastate power grid equipment beyond repair—with a file no bigger than a gif": https://www.wired.com/story/how-30-lines-of-code-blew-up-27-ton-generator/ https://www.wired.com/story/how-30-lines-of-code-blew-up-27-... YouTube video of demonstration: https://youtu.be/LM8kLaJ2NDU https://youtu.be/LM8kLaJ2NDU
- mtsr 6y agoBut you’re forgetting second order damage. What if somebody takes out a large enough fraction of power plants in the middle of winter? Or in the middle of a heat wave in Texas? Casualties could be rather significant if the plants can’t be restored in time.
- AnthonyMouse 6y ago> Your customers probably would be more willing to pay if they had to be insured against damages caused by hacking. This is actually the other way around. Insurance is a moral hazard. If you're paying for the insurance anyway then taking steps to reduce the insurance company's risk rather than your own is throwing money down the drain. Meanwhile the insurance company as a third party has less control over your actions than your own company and is less able to make efficient trade offs, so any requirements they impose tend to be either ineffective or unnecessarily burdensome or both. The problem is that publicly traded or municipal utility companies have pretty much the same problem where the employees/managers don't pay personally for damage and the people who do are too diffuse or far away to understand their exposure well enough to mitigate it.
- jacobwilliamroy 6y agoI will take the free government money thank you very much. Maybe even hire another person to help make these physical site visits. If we don't take the government's money, they'll just spend it on weapons.
- rhodozelia 6y agoPrivate and public companies own power plants too.
- jacobwilliamroy 6y agoCool. Let's lobby for regulations that force those companies to hire us. We're gonna make so much money.
- rhodozelia 6y agoYou’d make more money as a plumber
- kenneth 6y agoYou're essentially you're willing to trade security of critical infrastructure (and potentially risk lives) for the convenience of being able to do your job from home.
- rhodozelia 6y agoThe plant owners have made that call. Is a vpn secure? If Tesla can update the software in a fleet of vehicles remotely why can’t I update the software in a power plant?
- waste_monk 6y ago>The plant owners have made that call. Arguably it should not be up to the owner entity as to the security requirements they need to meet. At least down here in Australia we're in the process of legislative reforms that significantly expand both the scope of critical infrastructure (most of which is privately owned) and the responsibilities and requirements of critical infrastructure operators (in terms of risk management, security plans, mandatory reporting, etc.) >Is a vpn secure? Teamviewer (which I understand was the vector used in the water system breach) or a VPN that hasn't passed the extensive testing required of secure cryptographic devices is not acceptable. IF the industrial control systems were treated as a classified information system, were linked using an approved and properly managed [1] secure cryptographic device (on both sides e.g. back to a secure control centre, not to your house so you can work from bed), and generally had the rest of the protective elements (e.g. requiring personnel working on it to hold a SECRET or equivalent clearance proportionate with the value of the system) in place, that might be OK. But otherwise my position remains the same - it should not be remotely controllable via the internet. >If Tesla can update the software in a fleet of vehicles remotely why can’t I update the software in a power plant? That might brick some cars on-board computers or "only" kill perhaps a few dozen people before people caught on and it was recalled - still tragic, but small in the grand scheme of things. Critical infrastructure is called that because if it goes wrong it can harm or kill people on a vast scale, so it should be held to far higher standards. [1] e.g. keymat issued by whatever your equivalent to the NSA is e.g. our Australian Signals Directorate, and otherwise verified to be secure.
- hathawsh 6y agoThanks for speaking up, rhodozelia. The currently prevailing voice of HN appears not to be on your side, but you have 30 customers who are on your side, and you probably represent the views of many. We need to hear more of your side.
- rhodozelia 6y agoI wish more attention (money) was given to cyber security on infrastructure projects, and all projects. Cyber security is a mess everywhere. How Many organizations in the USA have not been hacked or are Unhackable? The only one I would guess is google and maybe amazon? Banks and financial markets seem competent as well.
- SXX 6y agoBanks generally have terrible track of record for security across the globe. But they usually have large and agressive legal departments so no sensible white hat ever gonna touch them with a ten-foot pole. At least they wont ever be allowed to publish results of security audit.
- eecc 6y agoSo did anyone of your deployments feature on one of the presentations of @viss - https://youtu.be/hMtu7vV_HmY https://youtu.be/hMtu7vV_HmY
- bryanrasmussen 6y agoso if the customers would pay you to drive to the middle of nowhere to make the change you still wouldn't do it?
- zhte415 6y agoI'm a bit more concerned why one person is critical for 30 power stations. Surely there should be at least a few others that could be called. What if you were on holiday, incapacitated or just quit?
- Communitivity 6y agoSCADA systems can kill people, any computerized system that can kill people must be protected as a weapons platform, therefore SCADA system must be protected as if they were weapons platforms. Wait, isn't that a bit over-dramatic? I mean sure, that's true of nuclear systems. But I mean, my home thermostat? My car's smart key system? As for the thermostat - it's winter below freezing and a hacker turns your heat off during the night. Sure someone might wake up, and it would have to be very cold for you to freeze, but if you have someone old or a child then it could be life threatening. Depending on the heater it might be able to make it cause a fire. Many car's smart key system is on the same network in the car as everything else, including the brakes. Read more in https://money.cnn.com/2014/08/01/technology/security/most-hackable-cars/index.html https://money.cnn.com/2014/08/01/technology/security/most-ha... So no, it's not being overly dramatic to categorize any SCADA system as a weapons platform. It might be, as they say, 'an abundance of caution', but I am fine with erring on that side than erring the other way. Three Mile Island can be viewed as caused because one gauge was not being tracked correctly (where it was supposed to be was being tracked, not where it actually was). What if remote access was enabled on systems in a nuclear power plant, so that people didn't have to drive all the way there? A hacker might manage to replicate some version of Three Mile Island. I can hear the 'but it's secured' response from some already, but any system that can be accessed can be hacked, given enough time and resources. The key thing people should realize is that cybersecurity does not prevent hacking, it attempts to delay it until a real person can take corrective action to protect the system. This is why you have defense in depth and rings of security. Essentially, it is a door lock. The better the lock, the more skilled class of thief it will dissuade by being 'too hard to bother'. A skilled thief who has state of the art lock picks, or is willing to use a door ram, is not going to be deterred by even the best of locks (think organized crime or state actor hackers here). But if our imaginary door only opens one way no matter what, to let people out, and can physically never let people in - then even the greatest thief is SOL. That is the purpose of a data diode. The other other concern, and it's not minor, is that any time you start measuring safety in terms of cost effectiveness you are putting a value on human life. It is saying that is is cheaper to pay the settlements and/or fines resulting from deaths due to insufficient safety precautions than it is to make it safe to begin with.
- bowmessage 6y agoThe data diode seems like an interesting idea, but couldn't that still enable a hacker (one who knew very well what they were doing) to increase the amount of lye in the water?
- waste_monk 6y agoIf you have the diode like: plant < diode < internet you could control it but not get any feedback. I meant plant > diode > internet so you could monitor it but not control it from the internet.
- bowmessage 6y agooops, duh, thanks for clarifying.
- closeparen 6y agoIt seems like a big jump from the kinds of idiotic security lapses we’re seeing to a full air gap. What’s wrong with proper authentication and encryption? It’s not a nuclear power plant. There’s no crack team of snipers to kill anything that gets too close to the fence. Physically these places are certainly not wide open, but neither are the fences, doors, and locks any more intense than what’s on the local high school.
- nicpottier 6y agoI think scale is the answer to that. These systems are being scanned for day and night for vulnerabilities from attackers and once they have infiltrated can lie in wait to do something whenever they want. A foreign actor can do that at massive scale undetected (both when successful and when not). Same isn't true for physical security.
- leghifla 6y agoWell, put too much inconvenience and some "smart" guy will circumvent it anyway. I have not personnaly worked in industrial environment, but from people I know, security wise, it is very scary at some places. Here is one such example of a "smart guy": The IT crew of a factory have a sane policy of deny all traffic, except some well thought out rules. A supplier would like to open a remote connection for maintenance of the equipment. Since the process for opening some traffic for a short duration is so convoluted, they finally install a 3G card inside the equipment (without telling their customer, and they even make it a habit for future install) so that they can do maintenance whenever they want. The IT crew still thinks all is safe from outside...
- TeMPOraL 6y agoThat's pretty much sabotage. Did someone in this story report it to the company management? This sounds like a clear lawsuit (or at the very least, contract termination) waiting to happen, and one that would scare other vendors away from such "clever" workarounds.
- leghifla 6y agoI don't know about management, but the guys on the factory floor were aware and happy with it. It made their job more convenient too, and the supplier was marketing its solutions as secure, so don't worry ;-)
- TeMPOraL 6y ago> the guys on the factory floor were aware and happy with it. It made their job more convenient too, Oh, I don't doubt that :). > the supplier was marketing its solutions as secure ... but I very much doubt the truth of that marketing :). I think it's still a ticking liability timebomb. At some point some external auditor will discover the backdoor to the plant and heads will roll.
- babas 6y agoThe 3G thing is extremely common. Even with HUGE suppliers. Including Siemens Energy. Their steam turbine systems come with a 3g connection as standard. I even think they won't do 24h/7d support without it.
- PicassoCTs 6y agoThis, this right here is the correct approach. Unfortunately, it is rarely taken, as economics are directly opposed to it. Usually you have a mixture of company wide LAN, with directly connected machines, accessible on demand via VPN from service-contractors. My fallback approach is, to insist on machines incapable of self-destruction and Whole System BackUps. Its easier & cheaper, to unplug, replace some spare-parts, reinstall some acronis images and go. Sad world, but real world