3 ms·

He is wrong and right. They should be connected to the Internet when they aren't 30 year old PLCs ripe for abuse. Until then, cut the data lines and do water mo

by 1970-01-01 3mo ago

He is wrong and right. They should be connected to the Internet when they aren't 30 year old PLCs ripe for abuse. Until then, cut the data lines and do water monitoring the old way.

grebc 3mo ago
If it’s connected, it’s compromised. Or will be. Folly to think otherwise.
- sublinear 3mo ago
  Keeping things up to date and holding critical infrastructure to higher standards than consumer tech is not a bad idea. Taking things offline and properly airgapped can also work, but wouldn't the cost of that exceed making specialized things and maintaining them? We got into this situation due to cost, not ignorance. Both choices are higher cost than putting ancient devices on the internet.
  - 27183 3mo ago
    You can't have both secure infrastructure and Internet exposed infrastructure. I don't know if I'd frame it as incompetence, but it does seem firmly outside the capabilities of current engineering practice. If you need a computer system to be actually secure, rule #0 is absolutely ensure it cannot receive unauthorized inputs of any kind (airgapped, big Faraday cage, JB Weld all the ports, big scary guys with guns, redundant locks, blast doors, etc). Otherwise you've lost against any sufficiently motivated adversary.
    - Kim_Bruning 3mo ago
      Right, enclose it in meters thick reinforced concrete walls and let no one near it. While that works for Chernobyl, if you have a real world systems you might want somewhat more practical access. Of course exposing industrial hardware directly on the internet is the other extreme, and you get what you're asking for. Do something in between, if you even just apply normal network security you'll be ahead of the pack. Problem is, a lot of these systems are not built by IT people. While they have a lot of quite admirable skills, it's just not their primary job, and thus they tend to lack the necessary paranoia at times.
      - 27183 3mo ago
        > normal network security The problem is there's no good way to actually enforce this. Every organization has their own idea of what is "good enough". The NSA has some pretty good advice[0]. But as far as I know there's no written-in-stone engineering standard organizations have to meet, just "best practices". If the building inspector finds fault with the construction of your facility, it gets evacuated and shut down until the defect is remedied. There's no inspector for your network security. That's the problem. [0] https://media.defense.gov/2022/Jun/15/2003018261/-1/-1/0/CTR_NSA_NETWORK_INFRASTRUCTURE_SECURITY_GUIDE_20220615.PDF https://media.defense.gov/2022/Jun/15/2003018261/-1/-1/0/CTR...
    - deleted 3mo ago
      [deleted]
  - oconnore 3mo ago
    I think you're assuming that updating software/hardware to more recent versions is sufficient to prevent a nation-state from wreaking havoc, and I'm not sure that's true. When it comes to something as critical as water infrastructure, maybe just don't connect the system to the internet on the off chance that you're wrong.
    - 27183 2mo ago
      In the case of a nation state actor, the most recent update could well be the attack vector.
  - Jtsummers 3mo ago
    > Keeping things up to date and holding critical infrastructure to higher standards than consumer tech is not a bad idea. This is a great theory, but practice (over centuries now if not millennia) tells us that critical infrastructure is rarely properly maintained. "If it ain't broke, don't fix it" is the motto of governments and large organizations everywhere when it comes to proper maintenance. As opposed to improper (keep the existing thing running) maintenance, proper maintenance requires being proactive and is expensive, often requiring partial or full replacements of systems while also keeping the old system running until a hand-off time. In order to get a government or corporation to be proactive, they have to see a problem. No problem, no worry. That it can be hacked is not a problem from their perspective. That it has been hacked might be a problem to them, but only if their constituents find out. More likely, they'll make it the poor engineer's problem, the engineer who had no budget and no staff to address it beforehand. When it's time to cut costs, proper maintenance is one of the first places organizations look to because it's not a present problem. Then it becomes normal to not do the work, from an organizational perspective, and all those engineers and technicians are just a bunch of Cassandras.
- lokar 2mo ago
  A lot of water infrastructure is physically spread out. It be very expensive and cumbersome (and probably inefficient) to require staff to by physically present at each site for monitoring and making any changes.
  - grebc 2mo ago
    That’s fine. It’s compromised. Happy with that?
    - lokar 2mo ago
      Or connect them securely
      - kelnos 2mo ago
        No such thing. There are levels of security, where higher levels mean more cost. And even with the most secure facility, there are no guarantees. And certainly we don't have the funding to provide that level of security for every single facility.
        lokar 2mo ago
        A well designed and deployed VPN link from the satellite location back to HQ (to machines isolated for just this use) would be orders of magnitude safer then what they seem to have been doing, and a reasonable compromise between security and utility.
        grebc 2mo ago
        It’s most definitely a compromise. Just not what you think. You can keep saying it’s more secure than currently, and that’s good and all. It still doesn’t matter against the adversaries being mentioned.
  - gopher_space 2mo ago
    I mean first you need to figure out the cost of a large municipality getting stuxnetted, and then you need to figure out where to hide when your constituents find out you made that kind of call.
eek2121 3mo ago
Disagree. Why connect them to the internet? They should be super hardened against attacks, and should NOT have a physical connection to the internet. Same with electrical infrastructure. Network access? Possibly, however that network should NOT be accessible from the internet. The only exception I can think of would be for meter reading, which should be a separate, read only device with no ability to do harm altogether.
- pixl97 3mo ago
  People start freaking out at the costs of dedicated fibers to every monitored facility. Hence even 'private' networks still run over the same actual lines as the internet.
  - baby_souffle 2mo ago
    The physical cost of installing definitely dominates the conversation but there is a reliability component as well. If you have a dedicated point-to-point fiber it's only one backhoe needed to ruin your day. If you have something like a tunnel over the internet, the death of one router or one link means you will probably just route around it and still be fine if you can tolerate the momentary blip in connectivity.
- snypher 2mo ago
  At a few of our sites, we have webcam-pointing-at-gauge. Deemed secure via air gap and simple to deploy.
  - 27183 2mo ago
    That's great! Different take on the opto-isolator.
Terr_ 3mo ago
Rather than "on/off" I think we need to distinguish between at least four things: 1. Connected naively to the internet. 2. Behind a hardened VPN endpoint which is on the internet. 3. Has a separate physical private network. 4. Requires physical access. I think it's obvious that #1 should be prohibited in favor of #2. After that point we need to ask what the impact is of a Denial of Service attack that prevents anyone from remotely accessing the system. The difference between #2 and #3 may depend on whether things could be Very Bad if the system is disconnected at a time of the attacker's choosing. For example, disabling access to flood-control valves during a hurricane.
- lokar 2mo ago
  The big question for 2 is what devices have access. If it’s a bunch of employees from loosely managed general use laptops, bad. If it’s a few computers at HQ that are totally locked down and without internet access, probably ok.
mikewarot 2mo ago
A data diode can allow monitoring via the internet without risking ingress of control. Commercial units aren't cheap though. Obviously we need open source designs. Perhaps the easiest way would be a Raspberry pi set up with an opto isolated CGA/EGA/VGA/SVGA capture that could be viewed via the internet? (I mean, we're probably talking systems still running MS-DOS or Windows 98 running these systems)
- gertrunde 2mo ago
  If you want to do that sort of thing in a cheap way, then you can. I'm told a box with two optical NICs in it with only one fibre in it, TX->RX, works quite well, plus a bit of software to wrap around it. But these sorts of organisations love a bit of bureaucratic regulation, and "certified" devices and that sort of thing, so there's probably not much of a market for a data diode without the paperwork. On the opto isolated idea - I think there was a QR-code and camera based data transfer tool posted on HN a few days ago!
  - bigfatkitten 2mo ago
    Data diode hardware isn’t terribly expensive, but the software runs into the hundreds of thousands precisely because these are niche products. To DIY it you need three NICs (or an optical isolator, which is honestly more work than it’s worth) because the transmitting NIC needs to see a link up before it transmits anything. That link is most easily provided by the second port on the transmitting side. Developing software to reliably move data across the link, and ensure that it’s properly formatted etc will still be your problem.
    - bigfatkitten 2mo ago
      Splitter, not isolator.
bigfatkitten 2mo ago
Things that don’t have a compelling need to be connected to the internet should never be connected to the internet. When there is a need, connectivity should be restricted to the bare minimum required for the task.

He is wrong and right. They should be connected to the Internet when they aren't 30 year old PLCs ripe for abuse. Until then, cut the data lines and do water monitoring the old way.

grebc 3mo ago

If it’s connected, it’s compromised. Or will be. Folly to think otherwise.

sublinear 3mo ago
Keeping things up to date and holding critical infrastructure to higher standards than consumer tech is not a bad idea. Taking things offline and properly airgapped can also work, but wouldn't the cost of that exceed making specialized things and maintaining them? We got into this situation due to cost, not ignorance. Both choices are higher cost than putting ancient devices on the internet.
- 27183 3mo ago
  You can't have both secure infrastructure and Internet exposed infrastructure. I don't know if I'd frame it as incompetence, but it does seem firmly outside the capabilities of current engineering practice. If you need a computer system to be actually secure, rule #0 is absolutely ensure it cannot receive unauthorized inputs of any kind (airgapped, big Faraday cage, JB Weld all the ports, big scary guys with guns, redundant locks, blast doors, etc). Otherwise you've lost against any sufficiently motivated adversary.
  - Kim_Bruning 3mo ago
    Right, enclose it in meters thick reinforced concrete walls and let no one near it. While that works for Chernobyl, if you have a real world systems you might want somewhat more practical access. Of course exposing industrial hardware directly on the internet is the other extreme, and you get what you're asking for. Do something in between, if you even just apply normal network security you'll be ahead of the pack. Problem is, a lot of these systems are not built by IT people. While they have a lot of quite admirable skills, it's just not their primary job, and thus they tend to lack the necessary paranoia at times.
    - 27183 3mo ago
      > normal network security The problem is there's no good way to actually enforce this. Every organization has their own idea of what is "good enough". The NSA has some pretty good advice[0]. But as far as I know there's no written-in-stone engineering standard organizations have to meet, just "best practices". If the building inspector finds fault with the construction of your facility, it gets evacuated and shut down until the defect is remedied. There's no inspector for your network security. That's the problem. [0] https://media.defense.gov/2022/Jun/15/2003018261/-1/-1/0/CTR_NSA_NETWORK_INFRASTRUCTURE_SECURITY_GUIDE_20220615.PDF https://media.defense.gov/2022/Jun/15/2003018261/-1/-1/0/CTR...
  - deleted 3mo ago
    [deleted]
- oconnore 3mo ago
  I think you're assuming that updating software/hardware to more recent versions is sufficient to prevent a nation-state from wreaking havoc, and I'm not sure that's true. When it comes to something as critical as water infrastructure, maybe just don't connect the system to the internet on the off chance that you're wrong.
  - 27183 2mo ago
    In the case of a nation state actor, the most recent update could well be the attack vector.
- Jtsummers 3mo ago
  > Keeping things up to date and holding critical infrastructure to higher standards than consumer tech is not a bad idea. This is a great theory, but practice (over centuries now if not millennia) tells us that critical infrastructure is rarely properly maintained. "If it ain't broke, don't fix it" is the motto of governments and large organizations everywhere when it comes to proper maintenance. As opposed to improper (keep the existing thing running) maintenance, proper maintenance requires being proactive and is expensive, often requiring partial or full replacements of systems while also keeping the old system running until a hand-off time. In order to get a government or corporation to be proactive, they have to see a problem. No problem, no worry. That it can be hacked is not a problem from their perspective. That it has been hacked might be a problem to them, but only if their constituents find out. More likely, they'll make it the poor engineer's problem, the engineer who had no budget and no staff to address it beforehand. When it's time to cut costs, proper maintenance is one of the first places organizations look to because it's not a present problem. Then it becomes normal to not do the work, from an organizational perspective, and all those engineers and technicians are just a bunch of Cassandras.
lokar 2mo ago
A lot of water infrastructure is physically spread out. It be very expensive and cumbersome (and probably inefficient) to require staff to by physically present at each site for monitoring and making any changes.
- grebc 2mo ago
  That’s fine. It’s compromised. Happy with that?
  - lokar 2mo ago
    Or connect them securely
    - kelnos 2mo ago
      No such thing. There are levels of security, where higher levels mean more cost. And even with the most secure facility, there are no guarantees. And certainly we don't have the funding to provide that level of security for every single facility.
      - lokar 2mo ago
        A well designed and deployed VPN link from the satellite location back to HQ (to machines isolated for just this use) would be orders of magnitude safer then what they seem to have been doing, and a reasonable compromise between security and utility.
        grebc 2mo ago
        It’s most definitely a compromise. Just not what you think. You can keep saying it’s more secure than currently, and that’s good and all. It still doesn’t matter against the adversaries being mentioned.
- gopher_space 2mo ago
  I mean first you need to figure out the cost of a large municipality getting stuxnetted, and then you need to figure out where to hide when your constituents find out you made that kind of call.

eek2121 3mo ago

Disagree. Why connect them to the internet? They should be super hardened against attacks, and should NOT have a physical connection to the internet. Same with electrical infrastructure. Network access? Possibly, however that network should NOT be accessible from the internet. The only exception I can think of would be for meter reading, which should be a separate, read only device with no ability to do harm altogether.

pixl97 3mo ago
People start freaking out at the costs of dedicated fibers to every monitored facility. Hence even 'private' networks still run over the same actual lines as the internet.
- baby_souffle 2mo ago
  The physical cost of installing definitely dominates the conversation but there is a reliability component as well. If you have a dedicated point-to-point fiber it's only one backhoe needed to ruin your day. If you have something like a tunnel over the internet, the death of one router or one link means you will probably just route around it and still be fine if you can tolerate the momentary blip in connectivity.
snypher 2mo ago
At a few of our sites, we have webcam-pointing-at-gauge. Deemed secure via air gap and simple to deploy.
- 27183 2mo ago
  That's great! Different take on the opto-isolator.

Terr_ 3mo ago

Rather than "on/off" I think we need to distinguish between at least four things: 1. Connected naively to the internet. 2. Behind a hardened VPN endpoint which is on the internet. 3. Has a separate physical private network. 4. Requires physical access. I think it's obvious that #1 should be prohibited in favor of #2. After that point we need to ask what the impact is of a Denial of Service attack that prevents anyone from remotely accessing the system. The difference between #2 and #3 may depend on whether things could be Very Bad if the system is disconnected at a time of the attacker's choosing. For example, disabling access to flood-control valves during a hurricane.

lokar 2mo ago
The big question for 2 is what devices have access. If it’s a bunch of employees from loosely managed general use laptops, bad. If it’s a few computers at HQ that are totally locked down and without internet access, probably ok.

mikewarot 2mo ago

A data diode can allow monitoring via the internet without risking ingress of control. Commercial units aren't cheap though. Obviously we need open source designs. Perhaps the easiest way would be a Raspberry pi set up with an opto isolated CGA/EGA/VGA/SVGA capture that could be viewed via the internet? (I mean, we're probably talking systems still running MS-DOS or Windows 98 running these systems)

gertrunde 2mo ago
If you want to do that sort of thing in a cheap way, then you can. I'm told a box with two optical NICs in it with only one fibre in it, TX->RX, works quite well, plus a bit of software to wrap around it. But these sorts of organisations love a bit of bureaucratic regulation, and "certified" devices and that sort of thing, so there's probably not much of a market for a data diode without the paperwork. On the opto isolated idea - I think there was a QR-code and camera based data transfer tool posted on HN a few days ago!
- bigfatkitten 2mo ago
  Data diode hardware isn’t terribly expensive, but the software runs into the hundreds of thousands precisely because these are niche products. To DIY it you need three NICs (or an optical isolator, which is honestly more work than it’s worth) because the transmitting NIC needs to see a link up before it transmits anything. That link is most easily provided by the second port on the transmitting side. Developing software to reliably move data across the link, and ensure that it’s properly formatted etc will still be your problem.
  - bigfatkitten 2mo ago
    Splitter, not isolator.

bigfatkitten 2mo ago

Things that don’t have a compelling need to be connected to the internet should never be connected to the internet. When there is a need, connectivity should be restricted to the bare minimum required for the task.