45 ms·
Hacker increased chemical level at Oldsmar's city water system, sheriff says
- hoppla 6y agoAccess Denied You don't have permission to access "http://www.wtsp.com/article/news/local/pinellascounty/pinellas-oldsmar-water-system-computer-intrustion/67-512b2bab-9f94-44d7-841e-5169fdb0a0bd http://www.wtsp.com/article/news/local/pinellascounty/pinell..." on this server. Reference #18.8700561.1612822078.194a07f6
- codetrotter 6y agoMaybe the hacker deleted the article too :^)
- amenghra 6y agohttps://twitter.com/zackwhittaker/status/1358867424171425799 https://twitter.com/zackwhittaker/status/1358867424171425799 has a link to a Youtube video if you want to watch the press conference.
- niea_11 6y agoProbably because you're accessing the website from the EU and they don't want to bother with the GDPR.
- MandieD 6y agoThis link works for me (EU): https://www.tampabay.com/news/pinellas/2021/02/08/someone-tried-to-poison-oldsmars-water-supply-during-hack-sheriff-says/ https://www.tampabay.com/news/pinellas/2021/02/08/someone-tr...
- dariusj18 6y agoThat looks like an Akamai error
- llacb47 6y agoThat's because Tegna is lazy and geoblocks EU visitors from their sites to avoid GDPR compliance.
- velox_io 6y agoYeah a lot of US news sites block Europe (and UK), a VPN is handy. Here's an unblocked source: https://www.vice.com/en/article/88ab33/hacker-poison-florida-water-pinellas-county https://www.vice.com/en/article/88ab33/hacker-poison-florida...
- rossdavidh 6y agoSo we are willing to let you access our water control systems, but not our news, basically. :)
- deleted 6y ago[deleted]
- amenghra 6y agofrom https://twitter.com/zackwhittaker/status/1358868187656388611 https://twitter.com/zackwhittaker/status/1358868187656388611: I can't immediately verify the veracity of the claims made by the sheriff but, the fact that the authorities *set up* a public-facing and/or remotely accessible system that allowed someone to change the water chemical levels is by far the bigger issue here.
- colechristensen 6y agoThere has been for quite a while a big concern that industrial control systems are accessible, often poorly hardened (and by that I mean to the extent of having default passwords), and quite vulnerable to attack. The only thing surprising about this is that we don't hear about it tenfold more.
- laurowyn 6y agoAbsolutely. Meanwhile, we live in a world where VPNs are sold to the casual user while critical systems are left on internet facing networks. I've never understood why, if these critical systems need remote access, it's not all done through a VPN of some sort. VPNs are not infallible, but it significantly increases the bar for entry from script kiddie to nation state real quick (depending on choice of crypt), while choosing a well supported implementation ensures long term bug fixes and security patches.
- bilbo0s 6y agoIn all honesty, why is a system so critical on the internet at all? People say, ease of administration, but there are other methods of achieving the same thing. Up to and including running your own network. On the one hand your engineers and chemists won't be able to fiddle with the aeration stage using their brand new, whiz bang, iPhone. On the other, the people in your community won't be put in harms way.
- the_only_law 6y ago
- ph4 6y agoWhy would 11,100 be recognized as a valid value to begin with?
- welder 6y agoYea, with the Mac key repeating issue that value could get input by accident. https://apple.stackexchange.com/questions/293523/single-keypress-types-repeated-characters-macbook-pro-2016-2017 https://apple.stackexchange.com/questions/293523/single-keyp... https://www.theverge.com/2020/5/4/21246223/macbook-keyboard-butterfly-magic-pro-apple-design https://www.theverge.com/2020/5/4/21246223/macbook-keyboard-...
- rootsudo 6y agoVs, why was this remote facing, and why don't they have a definitive answer on if it's an USA or non USA ip address. Sounds like no logs, probably showed up on shodan and someone wanted to have fun/many people did.
- mediocregopher 6y agoThe geolocation of the IP isn't all that useful, it could be a VPN or an owned machine.
- rootsudo 6y agoDisagree, if it was USA, it is easily possible to enforce a warrant and maybe you're lucky it's residential. If it was an VPN, you know it's a more competent person, org, and most VPN's also, keep logs.
- gruez 6y ago> Disagree, if it was USA, it is easily possible to enforce a warrant and maybe you're lucky it's residential. parent mentioned "owned machine" (as in, "hacked" not "ownership"), which means you might be able to find the source if you can seize the computer and analyze it in time. If the attacker wiped all traces from the computer then at best the trail ends there and at worst an innocent person gets blamed for it. >If it was an VPN, you know it's a more competent person, org, and most VPN's also, keep logs. "no log" is a commonly sought after feature in VPNs, and if you're planning to do shady stuff I doubt you'll go with a logged vpn.
- dialamac 6y agoLove that it was a “hacker” rather than a bored teenager or some casual idiot, or even a disgruntled former or current employee. Invoking the hacker term I guess is supposed to make it absolve gross negligence in even the most basic security practices. It makes me sad to think this spin actually works.
- amenghra 6y agoI thought we had all agreed that "Chinese hackers" was the correct term to use /s.
- nix23 6y agoYou forgot Russians/N.Koreans or Iranians...it's a flavor question.
- ogre_codes 6y agoThis was my gut instinct too. Some high school kid or just a random hacker who stumbled across it is just as likely as anything else.
- justaman 6y agoIf it calls attention to vulnerable infrastructure in the US, good. We need to put a lot more tax dollars into securing these systems.
- 6y ago
- _Microft 6y agoThe press conference can be found here: https://www.youtube.com/watch?v=MkXDSOgLQ6M https://www.youtube.com/watch?v=MkXDSOgLQ6M
- welder 6y agoI have an under-sink Reverse Osmosis water filter. If this wasn't caught I wonder if the RO filter would have removed the sodium hydroxide or not.
- leesalminen 6y agoEfficacy of RO systems is highly dependent on incoming water pressure and temperature. I got a professional water test done on our RO system and was surprised at how much remained in our (hard well) water. It turned out that our incoming water temp was far too low for the system to reach peak efficiency.
- Cerium 6y agoAnother aspect is TDS creep. The membrane only reaches the specified rejection rate after a few minutes of use. If they under sink RO system is used frequently for small amounts of water it can cause frequent cycling of the RO system which will reduce the water quality produced. My RO system can take the 450 TDS tap water down to about 30 under my normal use. If I close off the tank and run the water for about 10 minutes it will get down to about 20.
- welder 6y agoMine's tankless (Waterdrop brand), is TDS creep still an issue?
- extrapickles 6y agoYes, its a problem with the technology, so anything using RO will have the issue to some extent. Fancier systems will automatically flush the filter to help prevent this, but its rare on consumer grade equipment.
- welder 6y agoThe built-in TDS sensor shows 004 after filtering, but the unfiltered tap water is good here so that probably helps.
- beervirus 6y agoIf my vague memories of high school chemistry serve me correctly, then 11,100 ppm is 0.2775M, which would have a pH of about 13.4. That's definitely not something I'd want to drink.
- gowld 6y ago11K ppm is 1% of the entire water supply. I doubt the plant had that much lye in stock.
- beervirus 6y agoBut they probably had enough lye in stock to start producing water at that concentration.
- extrapickles 6y agoI hope they followed industry standards, and used a pump that was simply not able to supply harmful amounts of the chemical even when running at maximum speed.
- beervirus 6y agoSurely a water treatment plant that opens up controllers to the public Internet follows all industry standards!
- achillean 6y agoInternet-accessble industrial control systems have been a problem for many years now. It's a documented issue but it's difficult to fix for a variety of reasons: 1. Difficult to identify the owner: a lot of the devices are on mobile networks that don't point to an obvious owner. 2. Unknown criticality: is it a demo system or something used in production? 3. Security budget: lots of smaller utilities don't have a budget for buying cyber security products. 4. Uneducated vendor: sometimes the vendors of the device give very bad advice (https://blog.shodan.io/why-control-systems-are-on-the-internet/ https://blog.shodan.io/why-control-systems-are-on-the-intern...) That being said, based on the numbers in Shodan the situation has improved over the past decade. And there's been a large resurgence of startups in the ICS space (ex https://www.dragos.com https://www.dragos.com, https://www.gravwell.io https://www.gravwell.io). Here's a current view of exposed industrial devices on the Internet: https://beta.shodan.io/search/report?query=tag%3Aics&title=Industrial%20Control%20Systems%20Overview https://beta.shodan.io/search/report?query=tag%3Aics&title=I... I've written/ presented on the issue a few times: https://blog.shodan.io/taking-things-offline-is-hard/ https://blog.shodan.io/taking-things-offline-is-hard/ https://blog.shodan.io/trends-in-internet-exposure/ https://blog.shodan.io/trends-in-internet-exposure/ https://exposure.shodan.io/#/ https://exposure.shodan.io/#/
- giantg2 6y agoEven non-connected systems can be a problem. Stuxnet was an example. But I think the main point is that owners of those systems think they are protected just by being disconnected.
- nopzor 6y agothe stuxnet attack had a significantly higher level of sophistication than this. if your threat is a competent nation state the bar is much much higher.
- just_steve_h 6y agoIs the threat not from a competent nation-state or supranational entity? Is that not the intention of designating power & water & electricity systems as "critical infrastructure?"
- macawfish 6y agoOne thought that comes to mind: how do we know this same individual hasn't successfully gotten away with this elsewhere? How long would it take for people to report symptoms?
- throwawayboise 6y agoThere are QA checks on the "finished" water. Automated, and manual (i.e. a chemist manually sampling and analyzing the water). In this particular case, a high pH would have indicated that something was wrong and would have quickly been investigated before severe system-wide effects occurred.
- selimthegrim 6y agoCan confirm (former water treatment plant employee) but usually these are carried out first thing in the morning when the QA lab people get in as far as manually goes.
- billiam 6y agoIndividual? I don't see any evidence that points to whether it is a single, collective, state, private, or any other actor.
- technion 6y agoThis article specifically calls our Teamviewer as the vector: https://www.reuters.com/article/us-usa-cyber-florida-idUSKBN2A82FV https://www.reuters.com/article/us-usa-cyber-florida-idUSKBN...
- imiric 6y agoIt's baffling to me that TeamViewer is still used in a corporate setting, after all the vulnerabilities it's had over the years. This was nothing but gross negligence from whoever is in charge of their IT infrastructure.
- jishiav042 6y agoNot to mention Teamviewer is the go to software scammers in the name of tech support.
- probably_wrong 6y agoLet's not blame the knife company for the actions of a couple murderers. To put it in a different way: "Teamviewer: so reliable and easy to use than even your grandparents can install it."
- DevX101 6y agoFrom the article: "Thanks to a vigilant operator and several redundancies, the heightened level of sodium hydroxide never caused a public threat." I have a problem with the language here. This was absolutely a public threat. The attacker demonstrated intent and capability to inflict public harm. That's the definition of a threat. But the language downplaying the severity will mean this all blows over in a couple of months, without the actual mobilization/funds to properly secure not just this one site, but any similarly affected plants. I've come to the conclusion that humans in general aren't very good at preventing catastrophic events we haven't seen before (see climate change). We'll need to see n=1 disasters with this first, before there's public outcry to fix it.
- psadauskas 6y agoEven once isn't enough, or we learn the wrong lesson. Ask Flint, MI how well we've done fixing that catastrophic event.
- WillPostForFood 6y agoFor Americans who stopped following the Flint water crisis after its first few gritty chapters, it might come as a surprise how far the city has come: Today, after nearly $400 million in state and federal spending, Flint has secured a clean water source, distributed filters to all residents who want them, and laid modern, safe copper pipes to nearly every home in the city that needed them. Its water is as good as any city’s in Michigan. Sounds like they did a great job fixing it. https://www.politico.com/news/magazine/2020/12/23/flint-water-crisis-2020-post-coronavirus-america-445459 https://www.politico.com/news/magazine/2020/12/23/flint-wate...
- psadauskas 6y agoOnly took, what, 6 years? And still nobody has been held accountable?
- mc32 6y ago
- deleted 6y ago[deleted]
- rsync 6y agoWas this a social engineering attack wherein the attacker convinced an operator to work certain controls ? Certainly it is not conceivable that the controls themselves were connected to a network, right ? RIGHT?
- deleted 6y ago[deleted]
- cwkoss 6y ago> 100 -> 11100 I find it slightly amusing that it looks like the hacker just added two 1's to the front of a text box, rather than chose a specific value
- yalogin 6y agoThe whole town got really lucky. There was absolute intent to harm here, the attacker changed it from 100 to 10100, to fool a casual observer looking for the 100 pattern. We were told since the 9/11 time that our industrial control systems are in really bad shape, not sure if anything is done to strengthen it if at all. May be someone that's knowledgeable can chime in with information. I see a lot of scope for controls and operational procedure that can be streamlined and standardized across the whole country, if we have the will.
- sn_master 6y agoHow many times this could have happened without being detected?...
- ISL 6y agoAs someone who has some experience with hydroxide and water-treatment systems (as well as other potentially-dangerous industrial controls): always design your system such that even if your feed pump runs full-bore continuously, the system cannot harm anyone. The dilution of the solution stored in the hydroxide tank generally allows you to make this so. Sleeping well at night is a great side-effect.
- Kim_Bruning 6y ago* The motors are out to destroy the machine * The PLC is out to destroy the motors * SCADA/IPC is out to destroy the PLC Assuming these things in your design definitely helps with sound sleep. Especially when the company is running 3 shifts and you are on-call.
- lathiat 6y agoDeviant Ollam, well known presenter at various hacker conferences and professional penetration tester, has alluded to this a number of times in his presentations featuring photos of him in such water facilities: Featured in a few of these videos, hotlink to the slide: https://youtu.be/Rctzi66kCX4?t=2438 https://youtu.be/Rctzi66kCX4?t=2438
- nvader 6y agoIn the 90s there was a very campy TV Show called Superhuman Samurai. It was an Americanized Super Sentai show--think off-brand Power Rangers. The very scenario was the plot of one of the episodes. Here's a link, if anyone remembers the show and wants a dose of nostalgia. https://www.youtube.com/watch?v=TlJRumVVe10 https://www.youtube.com/watch?v=TlJRumVVe10
- userbinator 6y agoThe other question besides "why is this even on the Internet" is "why does this even need to be adjustable remotely from a computer?" Maybe I'm just really old-school, but it sounds like this sort of thing should really be something that's set once to the right values, and then if it ever needs changed, someone has to physically access a building and adjust a physical control --- likely alongside doing various other maintenance tasks on the system. This is different from remotely viewable, which is a much better idea, and I dare say should even be public.
- Johnny555 6y agoHow can they have remote access without any access controls and logging? an operator noticed someone had remotely entered the computer system that he was monitoring How could they not know immediately who it was (or at least, whose credentials were used)? Do they even know that it was a hacker and not someone trying to type "111.00 ppm but either they or the software dropped a decimal and typed "11100"?
- Johnny555 6y agoThey released a little more information -- they are apparently using some sort of remote desktop software for remote access: The intruder broke into the system at least twice on Friday, taking control of a plant operator's computer through the same methods a supervisor or specialist might use. The hack didn't initially set off red flags, because remote access is sometimes used to monitor the system or trouble-shoot problems, Gualtieri said So it almost certainly doesn't have enough auditing to know who made a change.
- titzer 6y agoThis is kind of horrifying. I find myself often coming to the conclusion that our increasing tangle of software dependencies is a massive liability to society. Suddenly code that was never designed to be "load-bearing"--intended to be part of a critical system, written in a hurry to glue together two components, or written for maximum performance and not maximum security, or code without proper tests, without quality control, code review, code written by hobbyists--finds itself into critical systems. And our amazing ability to constantly zoom out and plug and reuse software in huge amounts leads to this giant tangle of dependencies where everything is needed to get everything off the ground. And suddenly a JPEG vulnerability [1] leads to remote code execution. I don't know what the answer is, but I don't feel super great about the future, given the giant sinkhole of dependencies that everything seems to be getting sucked into. [1] https://www.techspot.com/community/topics/critical-vulnerability-found-in-in-jpeg-processing.15412/ https://www.techspot.com/community/topics/critical-vulnerabi...
- citrin_ru 6y agoThis one of main challenges in information security (and probably in IT in general) - code is completely invisible for a lay person and even for a professional it takes a lot of time to tell the difference between good and bad code. A hardware device, say a water pump made by industry leader would look very different from one made by a hobbyist in garage. It's obvious even to a lay person that a door from thick still is more break resistant than one from hardboard. But code written by a summer intern without any review and tests looks not much different from one written by seasoned professionals and carefully tested. Even if some software if full of RCE it takes a lot of time and motivation to find them. Huge amount of dependencies and many layers of abstractions make it even harder to see the software system as a whole.
- g_p 6y agoWhile horrifying, this is just the tip of the iceberg. Huge amounts of important infrastructure sits internet connected due to individual laziness, coupled with a lack of willingness to understand and think about cyber security. Often it seems simply from a lack of willingness to spend money on an ongoing basis to maintain anything. There's a culture and mindset in ICS that you don't change what isn't broken. And stability and reliability is important - this is an industry where you don't install patches due to the fear of breakage or regression. When the world shifted towards "code fast and break things", the ICS world didn't accept this change. They can't have Windows (yes, Windows) reboot unexpectedly to do an update. That pretty much rules out the supported versions like Windows 10. I mention consumer versions, because a culture of multi layer outsourcing means nobody wants to pay for a server version - an OEM Pro version of Windows saves a subcontractor some money. That OS won't be patched, unless the SCADA software vendor has validated the patch with the software being run. Expect crazy things like Windows XP SP2 (not 3) to be requirements. Everything is about stability and using tested configurations. You could be forgiven for thinking this is less scary, as you can airgap this, and treat it like a fixed appliance. Often that doesn't last, and (if you're lucky) an unpatched VPN box gets thrown in front of it with a weak password. More commonly, some consumer grade remote access software gets installed, so a bean counter can count how many beans they're making or spending. Airgap eliminated. The fix isn't single step - there's a need for more understanding about safety critical engineering in the IT world - the lack of testing and regression validation isn't acceptable to this industry. The ICS industry needs to be willing to pay for software maintenance and assured development processes. Simpler code that isn't running on full consumer operating systems is needed. And ultimately we need to go and replace systems that "ain't broke", but are insecure. And that's going to be expensive. No security appliances are needed here, just some basic common sense. Expect to see versions of Windows you didn't even know existed in use in very important places... Seeing pre-NT or very early NT wasn't a huge surprise...
- ck2 6y agoWhy the hell are these things on the internet? Why are power stations online? Are ICBMs on the internet? My gosh I hope not.
- deleted 6y ago[deleted]
- JohnJamesRambo 6y agoWhy does something like this control ever need to be online? I find myself saying that a lot these days.
- addicted 6y agoWhat evidence do we actually have that this was a hack and not maybe an accident or an inside job? It sounds really suspicious that the hack took the form of some sort of remote control which was evident to the actual operator who was present there. At the same time there was an actual operator, who wasn’t even suspicious the first time because apparently remote control was common by the supervisors. I think there’s a good chance we’re gonna find that either the operator, or one of the remote controllers accidentally, or maliciously, made this change, and blamed it on a “hack”.
- deleted 6y ago[deleted]
- ianhawes 6y agoMy first thought. While there is definitely major hype in infosec for ICS and other physical hacks, this sounds like either a disgruntled former employee or current employee going postal. Feds probably went along with this press conference for additional funding.
- tinmandespot 6y agoCoincidentally, I am reading the book Sandworm, which is about precisely this kind of threat to ICS from bad actors. I highly recommend it. https://www.penguinrandomhouse.com/books/597684/sandworm-by-andy-greenberg/ https://www.penguinrandomhouse.com/books/597684/sandworm-by-...
- DonHopkins 6y agoIsn't there a better term for a Hacker who pollutes water? Fracker?
- souprock 6y agoNaOH raised from 0.01% to 1.11% That seems to be a pH 13.4 result. There probably isn't much in the water to buffer that.
- marcosdumay 6y agoIt's baffling that the facility is even able to reach that number. There's absolutely no reason for a water treatment facility to put that much sodium hydroxide into water, so there is no reason to create hardware that can handle it.
- souprock 6y agoHow it could happen: Pump sizes can do 10, 20, 40, 80, 160, 320, 640, or 1280 units. Sometimes the water needs more chemicals. Maybe today it needs 10, but typical is 30, and it could go to 100. That's just the water rate for winter though. In summer, people wash cars and water lawns. Triple the number, so 100 becomes 300. The town is growing fast, at least when the equipment is installed. (thus the upgrade) In case the fast growth might continue, triple the number. The 300 becomes 900. Well, the smallest pump that will work is the one that does 1280 units. Also there is a federal grant from the EPA for big upgrades, and that one qualifies. Buy it.
- citrin_ru 6y agoAs long as who ever decided to connect critical infrastructure to the Internet is not held accountable we will hear more and more such stories. People are driven by incentives. There is a weak incentive for connecting system and practically none against.
- yodelshady 6y agoSide note: I thought the press conference (https://www.youtube.com/watch?v=MkXDSOgLQ6M https://www.youtube.com/watch?v=MkXDSOgLQ6M) was very well done. I'd differ on some minor matters of opinion, but overall Sheriff Gualtieri stuck to the facts, focused on the important matters, and generally treated the room with respect. Nice work.
- williesleg 6y agoWhy the fuck is the water system connected to the internet?
- peter303 6y agoJust outsource your cybersecurity to Solar Wind.
- thomas 6y agoWas the Pinella County Public Water Supply [0]? The article didn't make it clear. [0] https://mytapwater.org/pws/fl6521405/pinellas-county-utilities/pinellas-fl/ https://mytapwater.org/pws/fl6521405/pinellas-county-utiliti...