5 ms·

"In a perfect world, maybe there would be unlimited budgets for small rural water districts to have 24/7 onsite staff and run highly secured networks." I rejec

by rsync 6y ago

"In a perfect world, maybe there would be unlimited budgets for small rural water districts to have 24/7 onsite staff and run highly secured networks." I reject this line of thought. A small rural water district can run with looser tolerances and looser guarantees - and have done so for decades. They should spend half the time (and a quarter of the money) setting up systems that fail safely and revert to known states and operate with looser tolerances. As for telemetry ... I am not joking at all when I say that a green light on the building that turns red and everyone in the county knows to call either Jed or Billy if that light is red is a completely reasonable system. It's a small rural water district (your words) after all, right ?

sillysaurusx 6y ago
I invite you to become a pentester for a year and see if you still reject this line of thought.
- rsync 6y ago
  You misunderstand - I am saying they need neither the computers nor the networks. By running with loose tolerances and loose guarantees (and keeping systems as simple as possible) they remove the need for these tools - and their attack surface.
  - josefx 6y ago
    pen testing often involves walking into a building looking like you should be there. If no one is around to let you in it also can involve half a minute of getting the door open with some primitive every day tools. No computers or software required.
    - consp 6y ago
      You just switched the attacker model from "script kiddie somewhere in the world, changing stuff for the fun of it" to "physical presence with specific malicious intent". Those are not in the same ballpark, not even the same country.
      - josefx 6y ago
        You underestimate the stupid things local kids can get up to for fun if they thought they wouldn't get caught. So you can at least throw out the "specific malicious intent".
        elbear 6y ago
        Ok, but how many stupid local kids are there and how many worldwide script kiddies?
        heavenlyblue 6y ago
        Kids who are trespassing are not stupid. You have to take the risk, observe the behaviour of the security and then behave reasonably so that people would forgive you if you get caught. The action is stupid, but trespassing idiots get caught quickly - that's just "survival of the fittest" mechanics. If you never trespassed in your life then you were probably not smart enough to get away with it?
        ClumsyPilot 6y ago
        Security is a big word, we are talking about large facilities with like 1 or 2 people on site. As a kid I've tresspassed plenty, it often takes 'security' hours to even spot you
        codyb 6y ago
        I’m not particularly worried about a group of kids contaminating a single rural water supply, I’m more worried about coordinated effort by a malicious nation state to accumulate security cracks and then use them in a coordinated effort during a war time scenario.
    - mgreenleaf 6y ago
      I think it would be quite possible to have the control systems completely offline, while installing a reputable alarm system that is connected to the internet. If those two things have no network connection, then you could monitor the premises, but even a remote hacking of that system would not enable changing of chemical levels.
    - chopin 6y ago
      But you have this problem as well, even if you use networked systems. Now you need to mitigate both.
- philtar 6y ago
  You must be a pentester, because the commenter above you wasn't talking about pentesting at all. His comment says that small rural town water treatment plants don't have to run for 24/7. Not sure what you thought he was saying.
  - bryanrasmussen 6y ago
    >You must be a pentester, because the commenter above you wasn't talking about pentesting at all. This sounds like it should be a punchline to one of those funny programming jokes https://news.ycombinator.com/item?id=25850739 https://news.ycombinator.com/item?id=25850739
deleted 6y ago
[deleted]
ClumsyPilot 6y ago
It would make waay more sence for the entire state / country to buy software for their water systems all at once and set up a department of a few dozen people that travel the country and make sure its setup properly. Why would you not have all water treatment in the country on the same software platform?
- croon 6y ago
  A lot of these problems with inefficiency comes down to "muh state's rights", and less federal involvement. I think my stance shines through, but I understand it's debatable per case.
marcosdumay 6y ago
There is little problem on pushing telemetry into the internet. The largest issues is that you must gather the data from sensors that can't interfere with the thing you are measuring, and that you must process it with computers that don't connect to the ones controlling the process. The first one is really just good engineering practice, and the second is already cheap and getting cheaper by the day. Also, whatever you do at your process control, you should have some emergency overrides that set when the conditions get too abnormal. Those should be simple (AKA, no computers if possible) and stand-alone. Looks like they got this one right.
- discreteevent 6y ago
  Plus the benefit of setting something up to get telemetry as you described, is that someone won't later be tempted to use teamviewer (as in the article example) to open up the whole control system just to view some of that telemetery.
EvanAnderson 6y ago
re: alerting telemetry - I'm not finding photos to cite right now, but I've absolutely seen "infrastructure" buildings (here in rural Ohio) with warning annunciators (lights and bells) on their exteriors along with signs reading "If this light is flashing call xxx." It's definitely a viable system for alerting. I like your thinking and I try to espouse it myself (keeping things as simple as they can be-- keeping "technology" out of voting, not connecting things to networks that have no business being connected, etc). Short of a Battlestar Galactica-type "our machines rise up and try to kill us" event, though, I don't think the average person will ever understand the vulnerability inherent in networked computers or the risk/benefit tradeoff of connected vs. disconnected systems. Even down at the level of local politics in a rural setting the "optics" of bringing technological solutions to bear on problems is seen as forward-thinking-- particularly when it "saves" the taxpayer money. I can't imagine trying to convince a local water board that moving away from a PLC-based system with a remote support vendor world fly, even citing this example. This event will be another opportunity for more security vendors to cite in case studies justifying their products. More layers of garbage will build up on a foundation of protocols and design philosophies that grew up in an era of disconnected systems with lower stakes and a less complex threat model. I don't see big money to be made in providing sensible levels of connectivity and security to this kind of infrastructure. I don't see industry stepping-up because of that. Maybe regulation is the answer, though I'd just expect regulatory capture to take over, and have it become another "PCI". Maybe a lot of people have to die before society takes it seriously, as has been the case with so many other safety codes over human history. It makes me really sad, embarrassed for our industry, and more disappointed in human nature.
- mywittyname 6y ago
  > re: alerting telemetry - I'm not finding photos to cite right now, but I've absolutely seen "infrastructure" buildings (here in rural Ohio) with warning annunciators (lights and bells) on their exteriors along with signs reading "If this light is flashing call xxx." It's definitely a viable system for alerting. I can confirm these exist, as growing up my father's phone number was on a some of those signs. Fancy systems would automatically call an answering service, but for smaller ones, he'd get a call (or page, given the era), usually from a neighbor or something. It wasn't limited to rural areas either. I distinctly remember him taking service calls in ultra-wealthy neighborhoods on the weekends. One time, the gentleman who called in let my sister and I watch movies in his personal movie theater while my dad fixed the system.

by rsync 6y ago

sillysaurusx 6y ago

I invite you to become a pentester for a year and see if you still reject this line of thought.

rsync 6y ago
You misunderstand - I am saying they need neither the computers nor the networks. By running with loose tolerances and loose guarantees (and keeping systems as simple as possible) they remove the need for these tools - and their attack surface.
- josefx 6y ago
  pen testing often involves walking into a building looking like you should be there. If no one is around to let you in it also can involve half a minute of getting the door open with some primitive every day tools. No computers or software required.
  - consp 6y ago
    You just switched the attacker model from "script kiddie somewhere in the world, changing stuff for the fun of it" to "physical presence with specific malicious intent". Those are not in the same ballpark, not even the same country.
    - josefx 6y ago
      You underestimate the stupid things local kids can get up to for fun if they thought they wouldn't get caught. So you can at least throw out the "specific malicious intent".
      - elbear 6y ago
        Ok, but how many stupid local kids are there and how many worldwide script kiddies?
        heavenlyblue 6y ago
        Kids who are trespassing are not stupid. You have to take the risk, observe the behaviour of the security and then behave reasonably so that people would forgive you if you get caught. The action is stupid, but trespassing idiots get caught quickly - that's just "survival of the fittest" mechanics. If you never trespassed in your life then you were probably not smart enough to get away with it?
        ClumsyPilot 6y ago
        Security is a big word, we are talking about large facilities with like 1 or 2 people on site. As a kid I've tresspassed plenty, it often takes 'security' hours to even spot you
      - codyb 6y ago
        I’m not particularly worried about a group of kids contaminating a single rural water supply, I’m more worried about coordinated effort by a malicious nation state to accumulate security cracks and then use them in a coordinated effort during a war time scenario.
  - mgreenleaf 6y ago
    I think it would be quite possible to have the control systems completely offline, while installing a reputable alarm system that is connected to the internet. If those two things have no network connection, then you could monitor the premises, but even a remote hacking of that system would not enable changing of chemical levels.
  - chopin 6y ago
    But you have this problem as well, even if you use networked systems. Now you need to mitigate both.
philtar 6y ago
You must be a pentester, because the commenter above you wasn't talking about pentesting at all. His comment says that small rural town water treatment plants don't have to run for 24/7. Not sure what you thought he was saying.
- bryanrasmussen 6y ago
  >You must be a pentester, because the commenter above you wasn't talking about pentesting at all. This sounds like it should be a punchline to one of those funny programming jokes https://news.ycombinator.com/item?id=25850739 https://news.ycombinator.com/item?id=25850739

deleted 6y ago

[deleted]

ClumsyPilot 6y ago

It would make waay more sence for the entire state / country to buy software for their water systems all at once and set up a department of a few dozen people that travel the country and make sure its setup properly. Why would you not have all water treatment in the country on the same software platform?

croon 6y ago
A lot of these problems with inefficiency comes down to "muh state's rights", and less federal involvement. I think my stance shines through, but I understand it's debatable per case.

marcosdumay 6y ago

There is little problem on pushing telemetry into the internet. The largest issues is that you must gather the data from sensors that can't interfere with the thing you are measuring, and that you must process it with computers that don't connect to the ones controlling the process. The first one is really just good engineering practice, and the second is already cheap and getting cheaper by the day. Also, whatever you do at your process control, you should have some emergency overrides that set when the conditions get too abnormal. Those should be simple (AKA, no computers if possible) and stand-alone. Looks like they got this one right.

discreteevent 6y ago
Plus the benefit of setting something up to get telemetry as you described, is that someone won't later be tempted to use teamviewer (as in the article example) to open up the whole control system just to view some of that telemetery.

EvanAnderson 6y ago

re: alerting telemetry - I'm not finding photos to cite right now, but I've absolutely seen "infrastructure" buildings (here in rural Ohio) with warning annunciators (lights and bells) on their exteriors along with signs reading "If this light is flashing call xxx." It's definitely a viable system for alerting. I like your thinking and I try to espouse it myself (keeping things as simple as they can be-- keeping "technology" out of voting, not connecting things to networks that have no business being connected, etc). Short of a Battlestar Galactica-type "our machines rise up and try to kill us" event, though, I don't think the average person will ever understand the vulnerability inherent in networked computers or the risk/benefit tradeoff of connected vs. disconnected systems. Even down at the level of local politics in a rural setting the "optics" of bringing technological solutions to bear on problems is seen as forward-thinking-- particularly when it "saves" the taxpayer money. I can't imagine trying to convince a local water board that moving away from a PLC-based system with a remote support vendor world fly, even citing this example. This event will be another opportunity for more security vendors to cite in case studies justifying their products. More layers of garbage will build up on a foundation of protocols and design philosophies that grew up in an era of disconnected systems with lower stakes and a less complex threat model. I don't see big money to be made in providing sensible levels of connectivity and security to this kind of infrastructure. I don't see industry stepping-up because of that. Maybe regulation is the answer, though I'd just expect regulatory capture to take over, and have it become another "PCI". Maybe a lot of people have to die before society takes it seriously, as has been the case with so many other safety codes over human history. It makes me really sad, embarrassed for our industry, and more disappointed in human nature.

mywittyname 6y ago
> re: alerting telemetry - I'm not finding photos to cite right now, but I've absolutely seen "infrastructure" buildings (here in rural Ohio) with warning annunciators (lights and bells) on their exteriors along with signs reading "If this light is flashing call xxx." It's definitely a viable system for alerting. I can confirm these exist, as growing up my father's phone number was on a some of those signs. Fancy systems would automatically call an answering service, but for smaller ones, he'd get a call (or page, given the era), usually from a neighbor or something. It wasn't limited to rural areas either. I distinctly remember him taking service calls in ultra-wealthy neighborhoods on the weekends. One time, the gentleman who called in let my sister and I watch movies in his personal movie theater while my dad fixed the system.