6 ms·
Water system controllers don't belong on the internet, says ex-NSA chief
- mcfdoesdev 2mo agoI mean, this statement shouldn't be shocking to anyone. Critical infrastructure should be air-gapped if at all practical.
- Computer0 2mo ago[flagged]
- 1970-01-01 2mo agoHe is wrong and right. They should be connected to the Internet when they aren't 30 year old PLCs ripe for abuse. Until then, cut the data lines and do water monitoring the old way.
- grebc 2mo agoIf it’s connected, it’s compromised. Or will be. Folly to think otherwise.
- sublinear 2mo agoKeeping things up to date and holding critical infrastructure to higher standards than consumer tech is not a bad idea. Taking things offline and properly airgapped can also work, but wouldn't the cost of that exceed making specialized things and maintaining them? We got into this situation due to cost, not ignorance. Both choices are higher cost than putting ancient devices on the internet.
- 27183 2mo agoYou can't have both secure infrastructure and Internet exposed infrastructure. I don't know if I'd frame it as incompetence, but it does seem firmly outside the capabilities of current engineering practice. If you need a computer system to be actually secure, rule #0 is absolutely ensure it cannot receive unauthorized inputs of any kind (airgapped, big Faraday cage, JB Weld all the ports, big scary guys with guns, redundant locks, blast doors, etc). Otherwise you've lost against any sufficiently motivated adversary.
- Kim_Bruning 2mo agoRight, enclose it in meters thick reinforced concrete walls and let no one near it. While that works for Chernobyl, if you have a real world systems you might want somewhat more practical access. Of course exposing industrial hardware directly on the internet is the other extreme, and you get what you're asking for. Do something in between, if you even just apply normal network security you'll be ahead of the pack. Problem is, a lot of these systems are not built by IT people. While they have a lot of quite admirable skills, it's just not their primary job, and thus they tend to lack the necessary paranoia at times.
- 27183 2mo ago> normal network security The problem is there's no good way to actually enforce this. Every organization has their own idea of what is "good enough". The NSA has some pretty good advice[0]. But as far as I know there's no written-in-stone engineering standard organizations have to meet, just "best practices". If the building inspector finds fault with the construction of your facility, it gets evacuated and shut down until the defect is remedied. There's no inspector for your network security. That's the problem. [0] https://media.defense.gov/2022/Jun/15/2003018261/-1/-1/0/CTR_NSA_NETWORK_INFRASTRUCTURE_SECURITY_GUIDE_20220615.PDF https://media.defense.gov/2022/Jun/15/2003018261/-1/-1/0/CTR...
- deleted 2mo ago[deleted]
- oconnore 2mo agoI think you're assuming that updating software/hardware to more recent versions is sufficient to prevent a nation-state from wreaking havoc, and I'm not sure that's true. When it comes to something as critical as water infrastructure, maybe just don't connect the system to the internet on the off chance that you're wrong.
- 27183 2mo agoIn the case of a nation state actor, the most recent update could well be the attack vector.
- Jtsummers 2mo ago> Keeping things up to date and holding critical infrastructure to higher standards than consumer tech is not a bad idea. This is a great theory, but practice (over centuries now if not millennia) tells us that critical infrastructure is rarely properly maintained. "If it ain't broke, don't fix it" is the motto of governments and large organizations everywhere when it comes to proper maintenance. As opposed to improper (keep the existing thing running) maintenance, proper maintenance requires being proactive and is expensive, often requiring partial or full replacements of systems while also keeping the old system running until a hand-off time. In order to get a government or corporation to be proactive, they have to see a problem. No problem, no worry. That it can be hacked is not a problem from their perspective. That it has been hacked might be a problem to them, but only if their constituents find out. More likely, they'll make it the poor engineer's problem, the engineer who had no budget and no staff to address it beforehand. When it's time to cut costs, proper maintenance is one of the first places organizations look to because it's not a present problem. Then it becomes normal to not do the work, from an organizational perspective, and all those engineers and technicians are just a bunch of Cassandras.
- lokar 2mo agoA lot of water infrastructure is physically spread out. It be very expensive and cumbersome (and probably inefficient) to require staff to by physically present at each site for monitoring and making any changes.
- grebc 2mo agoThat’s fine. It’s compromised. Happy with that?
- lokar 2mo agoOr connect them securely
- kelnos 2mo agoNo such thing. There are levels of security, where higher levels mean more cost. And even with the most secure facility, there are no guarantees. And certainly we don't have the funding to provide that level of security for every single facility.
- lokar 2mo agoA well designed and deployed VPN link from the satellite location back to HQ (to machines isolated for just this use) would be orders of magnitude safer then what they seem to have been doing, and a reasonable compromise between security and utility.
- grebc 2mo agoIt’s most definitely a compromise. Just not what you think. You can keep saying it’s more secure than currently, and that’s good and all. It still doesn’t matter against the adversaries being mentioned.
- gopher_space 2mo agoI mean first you need to figure out the cost of a large municipality getting stuxnetted, and then you need to figure out where to hide when your constituents find out you made that kind of call.
- eek2121 2mo agoDisagree. Why connect them to the internet? They should be super hardened against attacks, and should NOT have a physical connection to the internet. Same with electrical infrastructure. Network access? Possibly, however that network should NOT be accessible from the internet. The only exception I can think of would be for meter reading, which should be a separate, read only device with no ability to do harm altogether.
- pixl97 2mo agoPeople start freaking out at the costs of dedicated fibers to every monitored facility. Hence even 'private' networks still run over the same actual lines as the internet.
- baby_souffle 2mo agoThe physical cost of installing definitely dominates the conversation but there is a reliability component as well. If you have a dedicated point-to-point fiber it's only one backhoe needed to ruin your day. If you have something like a tunnel over the internet, the death of one router or one link means you will probably just route around it and still be fine if you can tolerate the momentary blip in connectivity.
- snypher 2mo agoAt a few of our sites, we have webcam-pointing-at-gauge. Deemed secure via air gap and simple to deploy.
- 27183 2mo agoThat's great! Different take on the opto-isolator.
- Terr_ 2mo agoRather than "on/off" I think we need to distinguish between at least four things: 1. Connected naively to the internet. 2. Behind a hardened VPN endpoint which is on the internet. 3. Has a separate physical private network. 4. Requires physical access. I think it's obvious that #1 should be prohibited in favor of #2. After that point we need to ask what the impact is of a Denial of Service attack that prevents anyone from remotely accessing the system. The difference between #2 and #3 may depend on whether things could be Very Bad if the system is disconnected at a time of the attacker's choosing. For example, disabling access to flood-control valves during a hurricane.
- lokar 2mo agoThe big question for 2 is what devices have access. If it’s a bunch of employees from loosely managed general use laptops, bad. If it’s a few computers at HQ that are totally locked down and without internet access, probably ok.
- mikewarot 2mo agoA data diode can allow monitoring via the internet without risking ingress of control. Commercial units aren't cheap though. Obviously we need open source designs. Perhaps the easiest way would be a Raspberry pi set up with an opto isolated CGA/EGA/VGA/SVGA capture that could be viewed via the internet? (I mean, we're probably talking systems still running MS-DOS or Windows 98 running these systems)
- gertrunde 2mo agoIf you want to do that sort of thing in a cheap way, then you can. I'm told a box with two optical NICs in it with only one fibre in it, TX->RX, works quite well, plus a bit of software to wrap around it. But these sorts of organisations love a bit of bureaucratic regulation, and "certified" devices and that sort of thing, so there's probably not much of a market for a data diode without the paperwork. On the opto isolated idea - I think there was a QR-code and camera based data transfer tool posted on HN a few days ago!
- bigfatkitten 2mo agoData diode hardware isn’t terribly expensive, but the software runs into the hundreds of thousands precisely because these are niche products. To DIY it you need three NICs (or an optical isolator, which is honestly more work than it’s worth) because the transmitting NIC needs to see a link up before it transmits anything. That link is most easily provided by the second port on the transmitting side. Developing software to reliably move data across the link, and ensure that it’s properly formatted etc will still be your problem.
- bigfatkitten 2mo agoSplitter, not isolator.
- bigfatkitten 2mo agoThings that don’t have a compelling need to be connected to the internet should never be connected to the internet. When there is a need, connectivity should be restricted to the bare minimum required for the task.
- clbrmbr 2mo agoThere are many wireless pump-and-reservoir systems that while not internet connected, use insecure RF links. These local RF (and casting a wider net, Bluetooth) interfaces are also ripe for abuse.
- Wowfunhappy 2mo agoWouldn't the physical facilities themselves have security?
- lokar 2mo agoI assume they are talking about things like water towers that are spread around, and linked back to hq via insecure wireless.
- amluto 2mo agoYou can also find all kinds of interesting water infrastructure, often with no electronics, all around town. It rarely has any sort of security beyond a padlock.
- procarch2019 2mo agoYou’d be surprised how insecure some of these facilities are, especially to someone who has working knowledge of what a PLC (or other process controllers) does and how it works. You can easily look like a tech who belongs there either troubleshooting something or working on a project. I’ve been doing industrial controls for 15 years and surprisingly infrastructure is some of the most poorly funded. I believe a lot of these places are run by operating companies, so it’s bidded out (we all know how bids work I think). I’m not surprised when I walk into these places and see the computers are running EOL operating systems and the networking is essentially flat.
- Wowfunhappy 2mo agoOkay but then is the RF connection really your biggest concern? I'm kind of worried (probably unnecessarily) that posting ideas would get me on some list, but it seems like there would be many simpler terrorism opportunities once you have physical access.
- Cider9986 2mo ago>Other countries start securing their water >nsa: what no, stop that
- Terr_ 2mo agoI sometimes wonder how much damage (and potential damage) to US infrastructure exists simply because intelligence-agencies prioritize being able to exploit it globally over fixing it on defense.
- yellow_postit 2mo agoI sometimes think about how much the CIA risked setting back global health when using a fake vaccination program to find Bin Laden.
- deleted 2mo ago[deleted]
- psunavy03 2mo agoThe NSA's job is to collect information, not screw with people's water.
- rgblambda 2mo agoThe security vulnerabilities the NSA would use to spy on a system, or even just use as a jumping point to another system, can also be used by bad actors for other purposes, such as screwing with people's water.
- psunavy03 2mo agoAbsolutely true. It's just tiresome to hear internet people decide the NSA is $GENERIC_VILLAIN when they're an intelligence agency with a given remit. Of course their job is to hack and penetrate foreign systems to get intelligence information for the Federal government. That's well-documented. But that's not the same as claiming their job is to create chaos for $PARANOID_REASON. That's internet tinfoil-hattery.
- Kim_Bruning 2mo agoDon't put your PLCs directly on the internet. In fact most industrial stuff is very not made to be directly connected to the internet. But interpose a firewall+VPN solution and you might be ok, if done competently. And remote access to hardware definitely makes management and maintenance a lot easier and quicker. (else you need to drive out for every minor issue)
- tomsanbear 2mo ago"If done competently" is a bold assumption unfortunately, not just these days but always
- closeparen 2mo agoIf we can beat a security-state airgap by sprinkling USB drives in the parking lot, I think the Iranians can beat an "internal" network by getting someone to click on an email attachment or visit the wrong website on their municipal issued Windows machine. It's very common for the proprietary software for interfacing with ancient, expensive machines to break after OS upgrades, so they're probably unpatched... you might not even need to burn a 0-day.
- Kim_Bruning 2mo agoWhich would then tunnel through the VPN and still hit the PLC. Right. "But we need it on our windows computer" is not an entirely unreasonable ask. But security is only partially served.
- vannevar 2mo agoWe could say that about a lot of infrastructure that has been recklessly placed on the open Internet because it was cheaper than more secure solutions. I would say the same about home security systems, for instance.
- nik282000 2mo agoNo infrastructure should be on the open internet the potential for abuse is incredible. At a minimum a VPN should be used to tunnel all connections back to what ever command and control server exist, leaving equipment visible on the public IP should be a crime if not a felony.
- aliasxneo 2mo agoPart of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worked with an integrator who was working on an AHU feeding an extremely critical portion of a datacenter (I was a lead by this point and mostly played babysitter). During certain points of the day you couldn't open the door to this room due to negative pressure because the logic was over-ramping the exhaust fans. As I watched this contractor work, I saw him open his laptop, with Windows on it (because Microsoft has had a death grip on this industry for decades now), and proceed to backup the PLC program into a massive folder with God knows how many other "customer projects" he was carrying around in this thing. He then proceeded to go do some physical checks in the field, came back, and prepared to upload the fixed program. As I watched, I noticed he _grabbed a backup from ANOTHER customer_ and I immediately had to intervene. Who knows what untold damage I saved from that single move. I tell this story to demonstrate just how far into the dark ages this industry is. I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID and why they might want to consider a disaster recovery plan for their multi-million dollar mechanical plant. This industry is in _desperate_ need of strong technical help, but unfortunately the "higher ups" tend to be the same people who are "comfortable" with the way thing are and refuse to move. I literally tried for a decade before giving up and moving into software engineering proper. So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure.
- elevation 2mo agoA colleague of mine transitioned from a senior IC role to manager of a PLC group in the same company. Their methods were just as unsophisticated as you describe, and now he is attempting to evangelize/impose the finer parts of software engineering discipline, such as source control and integration tests (which can be trickier when it requires a hardware test bed.) Bringing some new tools to bear can be a fun job, as long as the team trusts you.
- jacobgold 2mo agoWith coding agents now being used for hacking, there's a decent chance we'll see a 9/11-scale hacking incident as a result of NSA/DHS negligence in securing American internet-connected services. Similar to how the CIA's negligence allowed 9/11 itself. The USG should be deploying thousands of security engineers armed with the latest coding models and agents, in attempt to secure systems before they're hacked. A few billion dollars spent here could save us trillions.
- bjt 2mo agoUnlike the CIA and/or FBI being in charge of preventing attacks like 9/11, NSA and DHS aren't in charge of each state's or city's infrastructure. They could run some opt-in program that local governments could then engage with, but "negligence" is a bit strong for systems they're not in charge of and have no access to.
- tony69 2mo ago“Not in charge of” can agree “Have no access to” I have doubts
- ungreased0675 2mo agohttps://www.nsa.gov/Cybersecurity/Cybersecurity-Advisories-Guidance/ https://www.nsa.gov/Cybersecurity/Cybersecurity-Advisories-G... While that’s not the job of the NSA, they do produce a lot of good cybersecurity guides.
- jacobgold 2mo agoNSA's job is SIGINT, so it's at least partly their responsibility when the threat is foreign. For example, the NSA should be successfully infiltrating every major foreign hacking team in the world and monitoring and/or disrupting their activities.
- judge2020 2mo ago> the NSA should be successfully infiltrating every major foreign hacking team in the world and monitoring and/or disrupting their activities. I'm sure this exists, but I'd expect CIA to be more involved in any sort of covert operations across nation state lines, especially if undercover humans are involved.
- mpfh 2mo ago[dead]
- fathermarz 2mo ago[dead]
- thisisnotauser 2mo agoDuh.
- gz5 2mo agowhat does belong on the internet in a post-mythos world? one argument: only services which need to be available to unauthenticated endpoints should be default reachable. all other services should be default unreachable (no data plane until authorized ...then use internet and other networks to establish the connections). yes, that is not always easy. it is much more possible than it used to be. and arguably we now need to commit to the tradeoffs of default unreachable services.
- Ozzie-D 2mo ago[flagged]
- abdullahk0634 2mo ago[dead]
- logicallee 2mo ago>Water system controllers don't belong on the internet, says ex-NSA chief sounds like cope for a bunch of felons that management, its director and congress can't get a handle on.
- chmod775 2mo agoAt modern population densities, basic infrastructure breaking on down on a large scale can kill millions in a matter of weeks. The largest threat isn't bombs falling on our heads, it's incompetent fools leaving the door open to their enemies. These aren't mistakes that can be excused. Failing in one's duty to steward important infrastructure must mean immediate replacement of leadership.
- wolvoleo 2mo agoWell yes this is true but air gapping isn't perfect security, it's important to understand that too. And true air gapping isn't possible because it'll need to be monitored somewhere central so there'll have to be some vpn or mpls whatever. Meaning it can be hacked. Having it exposed to the public internet is not good practice but should be far from the only layer in its security. And really, when a PLC is found unfirewalled on the public internet, you can bet that's far from the only security screwup in that infrastructure. If they won't even handle the low hanging fruit.
- nacozarina 2mo agoAnother vulnerability is natural gas pipeline compressor stations are all internet-connected, often via wireless because they are in remote locations. Simple DOS attack against one will force it to shutdown flow, causing things like power stations to drop offline.
- knorker 2mo agoNSA chief also says water is wet.