6 ms·
In a perfect world, maybe there would be unlimited budgets for small rural water districts to have 24/7 onsite staff and run highly secured networks. I regular
by ac29 6y ago
In a perfect world, maybe there would be unlimited budgets for small rural water districts to have 24/7 onsite staff and run highly secured networks.
I regularly work with these sorts of water districts (larger, better funded ones as well). In reality, some of these small districts may only have 2 or 3 SCADA operators on staff. Sending them home with a pager, a tablet, a VPN password, and some overtime pay is a lot easier to get past the city council then taking on another two employees to cover the night shift for those rare events that need to be handled ASAP.
I could share some real horror stories, but it wouldnt be professionally appropriate. Suffice to say, this story did not surprise me at all.
- nefitty 6y agoOf all the commentary I’ve read on this issue so far, this might be the scariest anecdote so far.
- joe_the_user 6y agoYou parent is saying: "just never put your stuff in the Internet". I suppose it really is "that hard" to tell clients this and we can expect more problem down the line.
- rsync 6y ago"In a perfect world, maybe there would be unlimited budgets for small rural water districts to have 24/7 onsite staff and run highly secured networks." I reject this line of thought. A small rural water district can run with looser tolerances and looser guarantees - and have done so for decades. They should spend half the time (and a quarter of the money) setting up systems that fail safely and revert to known states and operate with looser tolerances. As for telemetry ... I am not joking at all when I say that a green light on the building that turns red and everyone in the county knows to call either Jed or Billy if that light is red is a completely reasonable system. It's a small rural water district (your words) after all, right ?
- sillysaurusx 6y agoI invite you to become a pentester for a year and see if you still reject this line of thought.
- rsync 6y agoYou misunderstand - I am saying they need neither the computers nor the networks. By running with loose tolerances and loose guarantees (and keeping systems as simple as possible) they remove the need for these tools - and their attack surface.
- josefx 6y agopen testing often involves walking into a building looking like you should be there. If no one is around to let you in it also can involve half a minute of getting the door open with some primitive every day tools. No computers or software required.
- consp 6y agoYou just switched the attacker model from "script kiddie somewhere in the world, changing stuff for the fun of it" to "physical presence with specific malicious intent". Those are not in the same ballpark, not even the same country.
- deleted 6y ago[deleted]
- ClumsyPilot 6y agoIt would make waay more sence for the entire state / country to buy software for their water systems all at once and set up a department of a few dozen people that travel the country and make sure its setup properly. Why would you not have all water treatment in the country on the same software platform?
- croon 6y agoA lot of these problems with inefficiency comes down to "muh state's rights", and less federal involvement. I think my stance shines through, but I understand it's debatable per case.
- marcosdumay 6y agoThere is little problem on pushing telemetry into the internet. The largest issues is that you must gather the data from sensors that can't interfere with the thing you are measuring, and that you must process it with computers that don't connect to the ones controlling the process. The first one is really just good engineering practice, and the second is already cheap and getting cheaper by the day. Also, whatever you do at your process control, you should have some emergency overrides that set when the conditions get too abnormal. Those should be simple (AKA, no computers if possible) and stand-alone. Looks like they got this one right.
- discreteevent 6y agoPlus the benefit of setting something up to get telemetry as you described, is that someone won't later be tempted to use teamviewer (as in the article example) to open up the whole control system just to view some of that telemetery.
- EvanAnderson 6y agore: alerting telemetry - I'm not finding photos to cite right now, but I've absolutely seen "infrastructure" buildings (here in rural Ohio) with warning annunciators (lights and bells) on their exteriors along with signs reading "If this light is flashing call xxx." It's definitely a viable system for alerting. I like your thinking and I try to espouse it myself (keeping things as simple as they can be-- keeping "technology" out of voting, not connecting things to networks that have no business being connected, etc). Short of a Battlestar Galactica-type "our machines rise up and try to kill us" event, though, I don't think the average person will ever understand the vulnerability inherent in networked computers or the risk/benefit tradeoff of connected vs. disconnected systems. Even down at the level of local politics in a rural setting the "optics" of bringing technological solutions to bear on problems is seen as forward-thinking-- particularly when it "saves" the taxpayer money. I can't imagine trying to convince a local water board that moving away from a PLC-based system with a remote support vendor world fly, even citing this example. This event will be another opportunity for more security vendors to cite in case studies justifying their products. More layers of garbage will build up on a foundation of protocols and design philosophies that grew up in an era of disconnected systems with lower stakes and a less complex threat model. I don't see big money to be made in providing sensible levels of connectivity and security to this kind of infrastructure. I don't see industry stepping-up because of that. Maybe regulation is the answer, though I'd just expect regulatory capture to take over, and have it become another "PCI". Maybe a lot of people have to die before society takes it seriously, as has been the case with so many other safety codes over human history. It makes me really sad, embarrassed for our industry, and more disappointed in human nature.
- mywittyname 6y ago> re: alerting telemetry - I'm not finding photos to cite right now, but I've absolutely seen "infrastructure" buildings (here in rural Ohio) with warning annunciators (lights and bells) on their exteriors along with signs reading "If this light is flashing call xxx." It's definitely a viable system for alerting. I can confirm these exist, as growing up my father's phone number was on a some of those signs. Fancy systems would automatically call an answering service, but for smaller ones, he'd get a call (or page, given the era), usually from a neighbor or something. It wasn't limited to rural areas either. I distinctly remember him taking service calls in ultra-wealthy neighborhoods on the weekends. One time, the gentleman who called in let my sister and I watch movies in his personal movie theater while my dad fixed the system.
- windsurfer 6y agoIf all it takes to prevent the poisoning of an entire city's water supply is two employees, I certainly hope my governments are choosing to hire those two employees.
- tsimionescu 6y agoHow did they operate before computing became a mainstream thing? Utilities as a comcept are significantly older than computing itself, and certainly mainstream, internet-connected computation.
- deleted 6y ago[deleted]
- wnkrshm 6y agoThey had to pay more for employees who could be on site at unreasonable hours.
- tsimionescu 6y agoPerfect, unemployment is a major problem because of Covid19 anyway.
- redbeard0x0a 6y agoCOVID19 is also decimating budgets