5 ms·

from https://twitter.com/zackwhittaker/status/1358868187656388611 https://twitter.com/zackwhittaker/status/1358868187656388611: I can't immediately verify

from https://twitter.com/zackwhittaker/status/1358868187656388611 https://twitter.com/zackwhittaker/status/1358868187656388611: I can't immediately verify the veracity of the claims made by the sheriff but, the fact that the authorities *set up* a public-facing and/or remotely accessible system that allowed someone to change the water chemical levels is by far the bigger issue here.

colechristensen 6y ago
There has been for quite a while a big concern that industrial control systems are accessible, often poorly hardened (and by that I mean to the extent of having default passwords), and quite vulnerable to attack. The only thing surprising about this is that we don't hear about it tenfold more.
- laurowyn 6y ago
  Absolutely. Meanwhile, we live in a world where VPNs are sold to the casual user while critical systems are left on internet facing networks. I've never understood why, if these critical systems need remote access, it's not all done through a VPN of some sort. VPNs are not infallible, but it significantly increases the bar for entry from script kiddie to nation state real quick (depending on choice of crypt), while choosing a well supported implementation ensures long term bug fixes and security patches.
  - bilbo0s 6y ago
    In all honesty, why is a system so critical on the internet at all? People say, ease of administration, but there are other methods of achieving the same thing. Up to and including running your own network. On the one hand your engineers and chemists won't be able to fiddle with the aeration stage using their brand new, whiz bang, iPhone. On the other, the people in your community won't be put in harms way.
    - the_only_law 6y ago
      > On the one hand your engineers and chemists won't be able to fiddle with the aeration stage using their brand new, whiz bang, iPhone Maybe not at home, but couldn’t they have a local 802.11 network set up for this?
    - pseudoramble 6y ago
      No idea why this wasn’t. Often times they’re not for this reason _and_ because the hardware itself is too difficult or impossible to get online and can’t be upgraded. Forget networks for a moment - a system running Windows XP is way less risk than an upgrade to Windows 7. Plenty of companies have older systems running vital hardware that if it went offline could cause massive outages, revenue loss being only one if the impacts. So air gapped networks are pretty common in ICS environments as a result.
- JPKab 6y ago
  After meeting enough SAP consultants in the ICS space, all I can say is I'm shocked it doesn't happen every day.
- 2bitencryption 6y ago
  > The only thing surprising about this is that we don't hear about it tenfold more. If you're someone who stands to gain from disrupting a nation's infrastructure... you don't tip your hand until it most benefits you. If it really is the case that large parts of the infrastructure are very unsecure, expect to hear about it all at once, instead of little by little.
  - munk-a 6y ago
    Water seems like a really weird system to sabotage though - power can bring businesses offline in a serious way but a city reservoir likely isn't supplying any businesses with a real need of water for any sorts of industrial needs... It's more of an inconvenience. Messing with chemical balances in particular seems like a prank or someone really twisted trying to give a bunch of folks long term health complications.
    - devonkim 6y ago
      Sometimes attacks are probes and discoveries meant to determine or validate efficacy of a set of attack vectors including but not limited to human assets. Other uses are for distractions from other efforts. And yeah, sometimes they’re pranks. It’s not clear with the given facts what’s really going on.
    - vlovich123 6y ago
      You can't cause a rash of serious short term problems that increase the load on your health care system? That would be pretty compelling from a terrorist perspective or for nation states trying to demoralize/reduce trust in the current government.
    - TimothyBJacobs 6y ago
      Depends how long you can shut it down for. Even if only a few hours, an _unexpected_ shutdown of water across an entire city would certainly cause a panic with people descending on any stores open to buy water.
      - deleted 6y ago
- Analemma_ 6y ago
  To be honest, these days I just take it as a given that all critical US infrastructure (the power grid, hospitals, and now apparently water treatment plants) is riddled with time bombs, and that if we were to ever get in a shooting war with the countries which put them there, they'd all go off at once and we'd be in a world of hurt. I hope government/military planners are making the same assumption. It would be sort of darkly amusing if we've done the same thing to other countries, and so time bombs in infrastructure essentially replace nuclear weapons as the guarantors of Mutually Assured Destruction.
snypher 6y ago
Seems to me that the real issue is lack of security, not the fact this system exists at all. Eg Every cell tower has remote access protocol and we rarely hear about those being hacked.
- amenghra 6y ago
  There's probably 100x more cell towers than there are water plants. The impact of hacking a cell tower isn't direct loss of human life (granted, knowing off a large number of cell towers would be very disruptive). The answer to the question "should it be online" and "how much $$$ should we spend securing it" is going to be different in these two cases.
  - munk-a 6y ago
    I think there's also a fair question of "ownership of damages" here - cities get sold water treatment management systems and want them online as cheaply as possible - city councils end up owning the mistakes in misconfiguration but companies selling the systems are incentivized to make those default bad configurations possible - even while, in bold lettering, mentioning that you should not use the default authentication. Cell towers are a really integral part of carrier's business - I'm not certain whether most are owned by providers or other companies, but either way the folks that put the tower up owe the customer (be it a phone user, a phone provider or some subcontactor of the provider) an explanation and pay the costs of bad configuration... I'd also assume that making sure these towers stay up is someone's fulltime job (likely multiple people) - while there won't be an employee constantly monitoring city water systems since it would take so little of a single person's time.
    - Tyrek 6y ago
      I'm not sure I agree that this is /wrong/ per se - the issue arises from the city council's disinterest / lack of expertise (which itself comes from disinterest) in these systems. If the issues are disclosed clearly, and the city council continues to sign off on the implementation (due to disinterest, cost pressure, whatever) without consulting knowledgeable third parties, then it's only realistic that the blame falls on the ultimate decision-maker (in this case, the city council).
jcranmer 6y ago
I worked at a water treatment facility for a few summers, and the SCADA system there was on a physically separated network. Actually, there were two SCADA networks, one for each of the plants, with the distribution system (the water towers and pumping stations randomly scattered throughout the service area) attached to one of those networks. I don't know how secure those remote links were, but I suspect they were the easiest ingress into the network. A couple computers did bridge the two networks, but (IIRC) they were simple embedded systems doing read-only access (for compiling reports). I know when they did a pen-test, the pen-tester could compromise most of the corporate network (including service accounts), but they couldn't punch through to the SCADA systems.
- bilbo0s 6y ago
  I'm familiar with the systems you outline, and yes, those are more difficult to penetrate. However, those systems are significantly more expensive and more complex than the simpler ICS systems. Oldsmar Fl doesn't sound like a place that could afford such a system. Of course, can they afford not to have higher security systems is an open question?
  - koheripbal 6y ago
    The biggest cost of having physically separate networks (or at least network separated) is the HR cost of increased staffing and on-call requirements due to not being able to support the system remotely. For a small city, it's non-trivial.

colechristensen 6y ago

There has been for quite a while a big concern that industrial control systems are accessible, often poorly hardened (and by that I mean to the extent of having default passwords), and quite vulnerable to attack. The only thing surprising about this is that we don't hear about it tenfold more.

laurowyn 6y ago
Absolutely. Meanwhile, we live in a world where VPNs are sold to the casual user while critical systems are left on internet facing networks. I've never understood why, if these critical systems need remote access, it's not all done through a VPN of some sort. VPNs are not infallible, but it significantly increases the bar for entry from script kiddie to nation state real quick (depending on choice of crypt), while choosing a well supported implementation ensures long term bug fixes and security patches.
- bilbo0s 6y ago
  In all honesty, why is a system so critical on the internet at all? People say, ease of administration, but there are other methods of achieving the same thing. Up to and including running your own network. On the one hand your engineers and chemists won't be able to fiddle with the aeration stage using their brand new, whiz bang, iPhone. On the other, the people in your community won't be put in harms way.
  - the_only_law 6y ago
    > On the one hand your engineers and chemists won't be able to fiddle with the aeration stage using their brand new, whiz bang, iPhone Maybe not at home, but couldn’t they have a local 802.11 network set up for this?
  - pseudoramble 6y ago
    No idea why this wasn’t. Often times they’re not for this reason _and_ because the hardware itself is too difficult or impossible to get online and can’t be upgraded. Forget networks for a moment - a system running Windows XP is way less risk than an upgrade to Windows 7. Plenty of companies have older systems running vital hardware that if it went offline could cause massive outages, revenue loss being only one if the impacts. So air gapped networks are pretty common in ICS environments as a result.
JPKab 6y ago
After meeting enough SAP consultants in the ICS space, all I can say is I'm shocked it doesn't happen every day.
2bitencryption 6y ago
> The only thing surprising about this is that we don't hear about it tenfold more. If you're someone who stands to gain from disrupting a nation's infrastructure... you don't tip your hand until it most benefits you. If it really is the case that large parts of the infrastructure are very unsecure, expect to hear about it all at once, instead of little by little.
- munk-a 6y ago
  Water seems like a really weird system to sabotage though - power can bring businesses offline in a serious way but a city reservoir likely isn't supplying any businesses with a real need of water for any sorts of industrial needs... It's more of an inconvenience. Messing with chemical balances in particular seems like a prank or someone really twisted trying to give a bunch of folks long term health complications.
  - devonkim 6y ago
    Sometimes attacks are probes and discoveries meant to determine or validate efficacy of a set of attack vectors including but not limited to human assets. Other uses are for distractions from other efforts. And yeah, sometimes they’re pranks. It’s not clear with the given facts what’s really going on.
  - vlovich123 6y ago
    You can't cause a rash of serious short term problems that increase the load on your health care system? That would be pretty compelling from a terrorist perspective or for nation states trying to demoralize/reduce trust in the current government.
  - TimothyBJacobs 6y ago
    Depends how long you can shut it down for. Even if only a few hours, an _unexpected_ shutdown of water across an entire city would certainly cause a panic with people descending on any stores open to buy water.
    - deleted 6y ago
Analemma_ 6y ago
To be honest, these days I just take it as a given that all critical US infrastructure (the power grid, hospitals, and now apparently water treatment plants) is riddled with time bombs, and that if we were to ever get in a shooting war with the countries which put them there, they'd all go off at once and we'd be in a world of hurt. I hope government/military planners are making the same assumption. It would be sort of darkly amusing if we've done the same thing to other countries, and so time bombs in infrastructure essentially replace nuclear weapons as the guarantors of Mutually Assured Destruction.

snypher 6y ago

Seems to me that the real issue is lack of security, not the fact this system exists at all. Eg Every cell tower has remote access protocol and we rarely hear about those being hacked.

amenghra 6y ago
There's probably 100x more cell towers than there are water plants. The impact of hacking a cell tower isn't direct loss of human life (granted, knowing off a large number of cell towers would be very disruptive). The answer to the question "should it be online" and "how much $$$ should we spend securing it" is going to be different in these two cases.
- munk-a 6y ago
  I think there's also a fair question of "ownership of damages" here - cities get sold water treatment management systems and want them online as cheaply as possible - city councils end up owning the mistakes in misconfiguration but companies selling the systems are incentivized to make those default bad configurations possible - even while, in bold lettering, mentioning that you should not use the default authentication. Cell towers are a really integral part of carrier's business - I'm not certain whether most are owned by providers or other companies, but either way the folks that put the tower up owe the customer (be it a phone user, a phone provider or some subcontactor of the provider) an explanation and pay the costs of bad configuration... I'd also assume that making sure these towers stay up is someone's fulltime job (likely multiple people) - while there won't be an employee constantly monitoring city water systems since it would take so little of a single person's time.
  - Tyrek 6y ago
    I'm not sure I agree that this is /wrong/ per se - the issue arises from the city council's disinterest / lack of expertise (which itself comes from disinterest) in these systems. If the issues are disclosed clearly, and the city council continues to sign off on the implementation (due to disinterest, cost pressure, whatever) without consulting knowledgeable third parties, then it's only realistic that the blame falls on the ultimate decision-maker (in this case, the city council).

jcranmer 6y ago

I worked at a water treatment facility for a few summers, and the SCADA system there was on a physically separated network. Actually, there were two SCADA networks, one for each of the plants, with the distribution system (the water towers and pumping stations randomly scattered throughout the service area) attached to one of those networks. I don't know how secure those remote links were, but I suspect they were the easiest ingress into the network. A couple computers did bridge the two networks, but (IIRC) they were simple embedded systems doing read-only access (for compiling reports). I know when they did a pen-test, the pen-tester could compromise most of the corporate network (including service accounts), but they couldn't punch through to the SCADA systems.

bilbo0s 6y ago
I'm familiar with the systems you outline, and yes, those are more difficult to penetrate. However, those systems are significantly more expensive and more complex than the simpler ICS systems. Oldsmar Fl doesn't sound like a place that could afford such a system. Of course, can they afford not to have higher security systems is an open question?
- koheripbal 6y ago
  The biggest cost of having physically separate networks (or at least network separated) is the HR cost of increased staffing and on-call requirements due to not being able to support the system remotely. For a small city, it's non-trivial.