10 ms·

I don't have any specific knowledge of this event but I did work for a smart cities company for a while and can explain some of what hypothetically leads to mes

by kodah 6y ago

I don't have any specific knowledge of this event but I did work for a smart cities company for a while and can explain some of what hypothetically leads to messes like this. Most municipal infrastructure automation is run by SCADA systems. SCADA systems are designed to be closed ecosystems by default, which means they don't have access to the internet. Cables are run and point to point connections are made to transport these signals. If you've seen a big shiny aluminum box on the side of the road, that's a SCADA system. Consumer demand has changed a lot since SCADA was first thought up. These days people want to be able to log into a portal and see their billed water usage by the day. Cities want to be able to turn off a meter remotely at the click of a button rather than rolling a truck. They want pressure loss, contaminate, and fault detection. They'd like to balance pressure in the water system dynamically. They want to perform detailed reports on total water loss, which means instrumenting pumps, retrieving telemetry data, auditing that data against every home usage, identifying faults, and producing a report out of it. All of these things don't work very well with closed ecosystems as they generally require a certain amount of external information. In response, cities have either crafted their own devices because they don't have the money to hire an integrator to design and produce them a device that will survive extreme weather conditions, be fault tolerant, make minimal use of power, and employ standard security measures or they hire an integrator. Even certain integrators can do shotty work. The takeaway from this isn't that SCADA is bad or the infrastructure version of "the sky is falling!". It's that we desperately need to be thinking about how different generations of systems can be modularized and be built upon to minimize cost while maximizing security, capability growth, and reliability. For the folks talking about "who tests what", I don't know what I can talk about in regards to specific programs but I feel confident that I can say that the FBI tracks this vigorously and works with companies that work in public infrastructure.

EvanAnderson 6y ago
SCADA vendors have been way too caviler about collecting data from the "plant floor" for the "front office" with technologies that weren't designed with a threat model that even remotely approaches a run-of-the-mill LAN, let alone the Internet. Never mind protocols that have no provision for authentication or encryption-- we're talking about embedded devices that fall over and die when port-scanned or when they receive "too many" broadcasts. Network segmentation is usually difficult in these environments too. Inevitably, there's a piece of software and the associated devices it controls that have to layer 2-adjacent to "discover" each other and communicate. The SCADA vendor sets the expectation that a "front office" computer should be able to interface just fine with the "plant floor" and blames the meddling IT department with their subnets and default gateways and ACLs for things not working. (I may have some pent up frustration on this topic.) Aside from the technical limitations of some of these platforms there's a cultural issue. Controls engineers aren't trained in Infosec. The software and device manufacturers are developing better products, but I don't feel like there's a "secure by default" initiative. It still takes skills on the part of the installer. I still see a huge lack of consideration for security in the people actually installing gear (at least in the small-to-medium sized manufacturing companies and small municipalities I work around). The biggest concerns are making sure vendor's support personnel can troubleshoot remotely because frequently the on-site personnel have limited skills. That's how we end up with TeamViewer, VNC forwarded thru routers, etc.
- kodah 6y ago
  > The biggest concerns are making sure vendor's support personnel can troubleshoot remotely because frequently the on-site personnel have limited skills. That's an excellent call-out. City salaries will almost never employ the type and level of personnel to do anything beyond maintaining the status quo typically.
  - m4x 6y ago
    It's more than just a lack of salary. There simply isn't enough technical work to keep highly skilled engineers busy at your typical industrial or infrastructure sites. Even if you had the funds to employ highly trained engineers, most sites couldn't give them enough work to keep their skills sharp.
- ocdtrekkie 6y ago
  A lot of the failure is just the basics, and it's not always technical incompetence. It can be managerial. For instance, most SCADA and IT vendors can do a lot better than (probably not patched) Windows 7 with TeamViewer. However, there's two prevailing attitudes that interfere: 1. We're not big like Boston or New York, therefore we're not going to be targeted and are fine, nobody cares to harm us. 2. When you have a working SCADA system, nobody wants to run updates or make security policy changes when that might lead to not having a working SCADA system. Security vulnerability remains the constant invisible bogeyman, you never know if you did enough until it's too late.
  - dylan604 6y ago
    > 1. We're not big like Boston or New York, therefore we're not going to be targeted and are fine, nobody cares to harm us. If enough events keep occurring (and why would we suspect they won't), this attitude will HAVE?? to change. Kids will learn about it, and do it for the lulz. The city of Dallas had its tornado sirens hacked, and at least in the local areas, it brought some attention to the subject. Eventually, the bored teenagers in the small towns are going to do it just because.
waste_monk 6y ago
I've said it before and I'll say it again. The only way critical infrastructure should be connected to the internet is via data diode [1]. Otherwise it should be air-gapped. It still lets you remotely monitor systems, but if you want to adjust the operation of the critical infrastructure you have to physically go there and do it in person. People often complain that this is inefficient, but if your own workers are unable to remotely affect a system then neither can an attacker - given the consequences of an attacker compromising a critical infrastructure I think the inconvenience is a fair price to pay. [1] for those unfamiliar, this is typically implemented using two computers connected via fibreoptic cable, such that there are only transmitters (lasers) on one side and only receivers (photodiodes) on the other side. Thus the information can physically only travel in one direction. The computers at each end handle conversion from connection-oriented protocols (e.g. TCP) to connectionless (e.g. UDP) while maintaining reliability.
- rhodozelia 6y ago
  I work on about 30 different power plants for which I have written software in the control systems. I can access all of them remotely, no way i’m flying/Driving to middle of nowhere to make a change, and no way those customers are going to pay me for three days what otherwise takes a few hours.
  - usea 6y ago
    I worked on software that interfaced with industrial control systems on some large machinery that could definitely kill people (and sometimes did, anyway). We absolutely flew out to every location to make changes, around the world. Yes, it was a great cost of time and money. But it's just irresponsible to allow connections from the outside when something going wrong would kill people and cause a lot of damage.
    - rhodozelia 6y ago
      Other than syncing a generator totally out of phase I can’t think of anything that could potentially hurt someone at the power plants I work on. If the machines are killing people sometimes that is probably a whole other level of risk and I agree not something I would want to touch remotely.
dabbledash 6y ago
> Consumer demand has changed a lot since SCADA was first thought up. I find this hard to believe. In this modern world of connected convenience, I’d say what I want from the water company is 99.99999% safe water and 0.00001% real time billing. The idea that you would sacrifice the former for the latter is ridiculous. Nothing in your comment makes me think SCADA is bad, it just makes me think connecting it to the internet is dumb.
- ocdtrekkie 6y ago
  A big difference is employee and organizational expectations too. People expect to be able to monitor things from home, not pay people to sit at a desk and watch it.
  - dabbledash 6y ago
    That, I believe. But don’t try to wrap that in some nonsense about customers wanting it. What we want from water is safety. Period.
    - ocdtrekkie 6y ago
      I would also point out if you sell and support SCADA systems, the village is your customer/consumer, which I believe the parent post also alluded to.
    - at_a_remove 6y ago
      You are not the customer. You do not buy the SCADA system. You are the consumer. You want pure water. That I believe. The people buying the SCADA system want to be able to look at a dashboard while staying at home in their slippers during COVID-19. Worse yet, the SCADA system might have already been bought some time ago -- as is very common -- and adding "Hey I want to look at this from home!" might have happened a decade after the purchase.
    - kodah 6y ago
      As I noted, I worked for a smart cities company, the consumer for a SCADA and complimenting systems are both city employees and residents. You would also be surprised at what people desire out of their local water company. Portals and online payments were very much in demand. People don't like writing checks or having to go up to the municipal buildings to pay their bills. > What we want from water is safety. This is a different subject altogether. That said, the kind of pressure and pump monitoring systems I mentioned are all part of keeping water safe.
walrus01 6y ago
> These days people want to be able to log into a portal and see their billed water usage by the day. there's lots of safe ways to export data from a secured/highly controlled system in a manner designed for one way information flow. even if it's as simple as putting data in a pipe-delimited CSV in a cron job. first get the data out of the secured system's boundaries, nothing sensitive, and then do whatever is needed to present it to customers in a friendly web GUI.
netflixandkill 6y ago
The notion that SCADA systems are air gapped, closed ecosystems is 40 years out of date. I've had hundreds of plant engineers and SCADA operators tell me that their systems were "not connected to anything" and every single one of them was wrong. Whether it's an old 2400 baud modem on a spare port on the RTU or "serial" comms being multiplexed across IP or MPLS networks or the unlicensed TeamViewer install for vendor support, there is always a "well except ..."
MetalGuru 6y ago
Supervisory Control and Data Acquisition. Haven’t they always been able to remotely gather telemetry data and control devices? How was this achieved with network access?
Communitivity 6y ago
Great response. Sometimes they don't even craft devices, even shoddy devices would be better than other options some take. Sometimes it's just a matter of someone in the muni saying 'hey, we can fix this, let's put this on a password protected wireless network'. Sometimes it's not even password protected. Sometimes it's not the muni, but the vendor, who decides for remote management and puts the device on the internet. I agree with the other poster who said they thought data diode was the way to go. Make it so the companies and munis can get data, but cannot send data to the devices unless they are physically on site. Of course, that means the site has to be secured - a lot of times security through obscurity is relied on by putting in remote locales, fake locales, etc. This also includes securing the device - it should detect tampering and default to a 'tampered state' if tampered, one that is encoded at manufacturing so it cannot be changed. While I'm on it, I would love to start seeing TPM 2 chip and Sel4 OS a mandated minimal requirement for all SCADA devices. The cost would go through the roof - strike that, the atmosphere - but it is so necessary.

by kodah 6y ago

EvanAnderson 6y ago

SCADA vendors have been way too caviler about collecting data from the "plant floor" for the "front office" with technologies that weren't designed with a threat model that even remotely approaches a run-of-the-mill LAN, let alone the Internet. Never mind protocols that have no provision for authentication or encryption-- we're talking about embedded devices that fall over and die when port-scanned or when they receive "too many" broadcasts. Network segmentation is usually difficult in these environments too. Inevitably, there's a piece of software and the associated devices it controls that have to layer 2-adjacent to "discover" each other and communicate. The SCADA vendor sets the expectation that a "front office" computer should be able to interface just fine with the "plant floor" and blames the meddling IT department with their subnets and default gateways and ACLs for things not working. (I may have some pent up frustration on this topic.) Aside from the technical limitations of some of these platforms there's a cultural issue. Controls engineers aren't trained in Infosec. The software and device manufacturers are developing better products, but I don't feel like there's a "secure by default" initiative. It still takes skills on the part of the installer. I still see a huge lack of consideration for security in the people actually installing gear (at least in the small-to-medium sized manufacturing companies and small municipalities I work around). The biggest concerns are making sure vendor's support personnel can troubleshoot remotely because frequently the on-site personnel have limited skills. That's how we end up with TeamViewer, VNC forwarded thru routers, etc.

kodah 6y ago
> The biggest concerns are making sure vendor's support personnel can troubleshoot remotely because frequently the on-site personnel have limited skills. That's an excellent call-out. City salaries will almost never employ the type and level of personnel to do anything beyond maintaining the status quo typically.
- m4x 6y ago
  It's more than just a lack of salary. There simply isn't enough technical work to keep highly skilled engineers busy at your typical industrial or infrastructure sites. Even if you had the funds to employ highly trained engineers, most sites couldn't give them enough work to keep their skills sharp.
ocdtrekkie 6y ago
A lot of the failure is just the basics, and it's not always technical incompetence. It can be managerial. For instance, most SCADA and IT vendors can do a lot better than (probably not patched) Windows 7 with TeamViewer. However, there's two prevailing attitudes that interfere: 1. We're not big like Boston or New York, therefore we're not going to be targeted and are fine, nobody cares to harm us. 2. When you have a working SCADA system, nobody wants to run updates or make security policy changes when that might lead to not having a working SCADA system. Security vulnerability remains the constant invisible bogeyman, you never know if you did enough until it's too late.
- dylan604 6y ago
  > 1. We're not big like Boston or New York, therefore we're not going to be targeted and are fine, nobody cares to harm us. If enough events keep occurring (and why would we suspect they won't), this attitude will HAVE?? to change. Kids will learn about it, and do it for the lulz. The city of Dallas had its tornado sirens hacked, and at least in the local areas, it brought some attention to the subject. Eventually, the bored teenagers in the small towns are going to do it just because.

waste_monk 6y ago

I've said it before and I'll say it again. The only way critical infrastructure should be connected to the internet is via data diode [1]. Otherwise it should be air-gapped. It still lets you remotely monitor systems, but if you want to adjust the operation of the critical infrastructure you have to physically go there and do it in person. People often complain that this is inefficient, but if your own workers are unable to remotely affect a system then neither can an attacker - given the consequences of an attacker compromising a critical infrastructure I think the inconvenience is a fair price to pay. [1] for those unfamiliar, this is typically implemented using two computers connected via fibreoptic cable, such that there are only transmitters (lasers) on one side and only receivers (photodiodes) on the other side. Thus the information can physically only travel in one direction. The computers at each end handle conversion from connection-oriented protocols (e.g. TCP) to connectionless (e.g. UDP) while maintaining reliability.

rhodozelia 6y ago
I work on about 30 different power plants for which I have written software in the control systems. I can access all of them remotely, no way i’m flying/Driving to middle of nowhere to make a change, and no way those customers are going to pay me for three days what otherwise takes a few hours.
- usea 6y ago
  I worked on software that interfaced with industrial control systems on some large machinery that could definitely kill people (and sometimes did, anyway). We absolutely flew out to every location to make changes, around the world. Yes, it was a great cost of time and money. But it's just irresponsible to allow connections from the outside when something going wrong would kill people and cause a lot of damage.
  - rhodozelia 6y ago
    Other than syncing a generator totally out of phase I can’t think of anything that could potentially hurt someone at the power plants I work on. If the machines are killing people sometimes that is probably a whole other level of risk and I agree not something I would want to touch remotely.

dabbledash 6y ago

> Consumer demand has changed a lot since SCADA was first thought up. I find this hard to believe. In this modern world of connected convenience, I’d say what I want from the water company is 99.99999% safe water and 0.00001% real time billing. The idea that you would sacrifice the former for the latter is ridiculous. Nothing in your comment makes me think SCADA is bad, it just makes me think connecting it to the internet is dumb.

ocdtrekkie 6y ago
A big difference is employee and organizational expectations too. People expect to be able to monitor things from home, not pay people to sit at a desk and watch it.
- dabbledash 6y ago
  That, I believe. But don’t try to wrap that in some nonsense about customers wanting it. What we want from water is safety. Period.
  - ocdtrekkie 6y ago
    I would also point out if you sell and support SCADA systems, the village is your customer/consumer, which I believe the parent post also alluded to.
  - at_a_remove 6y ago
    You are not the customer. You do not buy the SCADA system. You are the consumer. You want pure water. That I believe. The people buying the SCADA system want to be able to look at a dashboard while staying at home in their slippers during COVID-19. Worse yet, the SCADA system might have already been bought some time ago -- as is very common -- and adding "Hey I want to look at this from home!" might have happened a decade after the purchase.
  - kodah 6y ago
    As I noted, I worked for a smart cities company, the consumer for a SCADA and complimenting systems are both city employees and residents. You would also be surprised at what people desire out of their local water company. Portals and online payments were very much in demand. People don't like writing checks or having to go up to the municipal buildings to pay their bills. > What we want from water is safety. This is a different subject altogether. That said, the kind of pressure and pump monitoring systems I mentioned are all part of keeping water safe.

walrus01 6y ago

> These days people want to be able to log into a portal and see their billed water usage by the day. there's lots of safe ways to export data from a secured/highly controlled system in a manner designed for one way information flow. even if it's as simple as putting data in a pipe-delimited CSV in a cron job. first get the data out of the secured system's boundaries, nothing sensitive, and then do whatever is needed to present it to customers in a friendly web GUI.

netflixandkill 6y ago

The notion that SCADA systems are air gapped, closed ecosystems is 40 years out of date. I've had hundreds of plant engineers and SCADA operators tell me that their systems were "not connected to anything" and every single one of them was wrong. Whether it's an old 2400 baud modem on a spare port on the RTU or "serial" comms being multiplexed across IP or MPLS networks or the unlicensed TeamViewer install for vendor support, there is always a "well except ..."

MetalGuru 6y ago

Supervisory Control and Data Acquisition. Haven’t they always been able to remotely gather telemetry data and control devices? How was this achieved with network access?

Communitivity 6y ago

Great response. Sometimes they don't even craft devices, even shoddy devices would be better than other options some take. Sometimes it's just a matter of someone in the muni saying 'hey, we can fix this, let's put this on a password protected wireless network'. Sometimes it's not even password protected. Sometimes it's not the muni, but the vendor, who decides for remote management and puts the device on the internet. I agree with the other poster who said they thought data diode was the way to go. Make it so the companies and munis can get data, but cannot send data to the devices unless they are physically on site. Of course, that means the site has to be secured - a lot of times security through obscurity is relied on by putting in remote locales, fake locales, etc. This also includes securing the device - it should detect tampering and default to a 'tampered state' if tampered, one that is encoded at manufacturing so it cannot be changed. While I'm on it, I would love to start seeing TPM 2 chip and Sel4 OS a mandated minimal requirement for all SCADA devices. The cost would go through the roof - strike that, the atmosphere - but it is so necessary.