4 ms·

This is mind boggling. Who installs these systems? Who maintains them? Surely this is supposed to be done by someone with at least a certain amount of clue? Eno

This is mind boggling. Who installs these systems? Who maintains them? Surely this is supposed to be done by someone with at least a certain amount of clue? Enough clue not to hook up your insecure gear to the internet? No? I don't even get how this happens. Surely these things are not just plugged in to a modem? There has got to be some kind of LAN involved. If there is, then there should at least be an edge firewall. Or even a simple garden variety gateway with NAT, which would already prevent all of those open ports from being accessible. So what gives? Are people deliberately hooking this gear up to the internet, deliberately exposing ports without taking security into consideration? That is patently insane.

TeMPOraL 7y ago
> Surely this is supposed to be done by someone with at least a certain amount of clue? Enough clue not to hook up your insecure gear to the internet? No? The way I understand it (and what my brief experience in writing software for industrial use confirms), it's the usual: the people making purchase decision have little to no clue, which - in absence of regulations - allows companies selling these products and services to not care, employ people without clue, and/or purposefully trade off quality for sales points (think of how many pseudo-features you can add to a PLC once you hook it up to the cloud).
tonyarkles 7y ago
My (not vast) experience with this stuff is that it’ll often get plugged directly into an LTE-Ethernet modem. Or, depending on the vintage of the equipment, the chain looks more like LTE-Ethernet modem connected to a “serial port concentrator” with Ethernet on one side and a bunch of serial ports on the other side. Sometimes the installers will turn on IP Whitelisting on the inbound ports, but sometimes that gets skipped.
janci 7y ago
I don't know about ICS, but CCTV cameras, HVAC controls, parking gates are too often installed by technicians with little to no IT knowledge. They ask the local IT to open port (ie. port forwarding) for remote management or servicing without knowing the implications. When offered a VPN they often refuse - they do not want or know how to use it.
noir_lord 7y ago
It’s not deliberate in the sense they want people to be vulnerable. It’s the path of least resistance in action. If you make it significantly more work to do something than not then people simply won’t do it. As the visible difference between secure and insecure is invisible to management it slides. As an industry we’ve failed all horribly at making secure the easy default option. And that’s going to haunt all of us.
- bsder 7y ago
  > As an industry we’ve failed all horribly at making secure the easy default option. This will never and can never happen. Security and sharing are at odds. If I don't want connectivity, I don't even buy a connected controller as it is more expensive. So, the fact that a connected controller got bought means someone wanted connectivity. Of course, the person who wanted connectivity didn't allocate any actual money to the person actually doing configuration, maintenance, auditing, updating, etc. Only at this point, do we have the fact that configuring a secured device is a pain. So, the default is to turn off all the security to get it up and running. After that, if someone can't tweak the software to turn the security back on in a couple of hours, it gets left off (remember--nobody actually allocated budget so these are unbillable, wasted hours). All of this together collides into a nice big pile of fail.
  - lazulicurio 7y ago
    > > As an industry we’ve failed all horribly at making secure the easy default option. > This will never and can never happen. Security and sharing are at odds I don't think the GP was suggesting that products be made so secure to be functionally useless. There's a wide spectrum between completely open and completely locked down. But right now the gradient to go from open -> secure is so steep that most people aren't willing to put in the effort. For example, have you ever tried using OpenSSL to create a self-signed certificate? The defaults are probably not what you want. Adding on stuff like the proper certificate extensions is even more work. Products should be designed so that it's not a herculean task to add security when you're ready to take that step.
- lazulicurio 7y ago
cookiecaper 7y ago
How much do engineers specializing in these systems get paid? I interviewed at a couple of places that do industrial-style systems and in one case, I got an offer that clocked in far below the requested rate (not a negotiation tactic, they declined to revise it), and at another, the recruiter couldn't hang up fast enough when she heard salary requirements. It's something like the Idiocracy Effect. In Idiocracy, the present-day world's best minds devote all their energy to eliminating hair loss and prolonging erections, eventually leading to a collapsed society. The real world is not too far off. Our best minds are dedicated to improving ad targeting and keeping users in vanity-gratifying loops on social media, while our industrial control systems that literally run the modern world wallow in the 70s. Everyone has some culpability in that. How many of us yearn to work for the local power utility (or one of their upstream vendors) rather than Facebook, Google, or the latest VC pump-and-dump scheme -- err, I mean, hot SV unicorn? Even if they did compensate competitively, are there enough hackers with a sense of [realism/duty/patriotism/INSERT_OTHER_SOCIAL_VALUE_HERE] to heed the call? There's definitely a role for some type of regulation or standards here. Industrial controls should be considered vital infrastructure that require serious and immediate investment. A brief visit to Shodan will show thousands of exposed industrial resources, and that should scare your pants off.
- ChuckNorris89 7y ago
  Don't know about the US market but in EU salaries in the embedded/automotive/industrial control are crap compared to frontend/app kiddies. It's definitely a tragedy of the commons, idiocracy style. Lots of embedded work related to critical infrastructure is now dumped to Eastern Europe or Asia as the good people in this field are mostly gray beards which are deemed too expensive by management.
- coryrc 7y ago
  I used to work on those sorts of systems. I'm now making 3x as much at FAANG. I love working on systems where you own it top-to-bottom, but not going to turn down extra piles of cash every month.
- nitrogen 7y ago
  How many of us yearn to work for the local power utility I actually would (or a vendor as you say) if it didn't mean earning 1/3 or less as much money. I mean, what better way to give back to the world than to apply one's hard-earned skills to improving the technologies (water, gas, power, manufacturing) that make modern society possible? It's like real life Ark or Infinifactory. But the pay is just not viable, and there are often stricter credential requirements than in valley software engineering.
jacquesgt 7y ago
I used to work in the industrial controls industry. The systems are often designed by application engineers working for the industrial control equipment’s manufacturer or distributor. In the case of distributors, the engineering work is often provided for “free” and paid for with the markup the distributor applies over their cost to purchase the components direct from the manufacturer. Those same application engineers will be involved with helping to make the sale. If a customer asks “can I connect this to the Internet?”, any response other than “of course!” is liable to result in a talking to from the sales manager for that account. Few of the incentives in that industry lead to good security. More details here: https://news.ycombinator.com/item?id=3260127 https://news.ycombinator.com/item?id=3260127
snowwindwaves 7y ago
Security is not mentioned in the specification for the control system so the company building it does nothing. When asked customer does not want to pay any more for secure remote access.
antisemiotic 7y ago
I can't say much about Switzerland, but on a slightly related note, in Poland we have automatic doors in trains that don't bother detecting if someone's in the doorway before closing. I'm honestly surprised that these didn't kill a single child yet.

TeMPOraL 7y ago

> Surely this is supposed to be done by someone with at least a certain amount of clue? Enough clue not to hook up your insecure gear to the internet? No? The way I understand it (and what my brief experience in writing software for industrial use confirms), it's the usual: the people making purchase decision have little to no clue, which - in absence of regulations - allows companies selling these products and services to not care, employ people without clue, and/or purposefully trade off quality for sales points (think of how many pseudo-features you can add to a PLC once you hook it up to the cloud).

tonyarkles 7y ago

My (not vast) experience with this stuff is that it’ll often get plugged directly into an LTE-Ethernet modem. Or, depending on the vintage of the equipment, the chain looks more like LTE-Ethernet modem connected to a “serial port concentrator” with Ethernet on one side and a bunch of serial ports on the other side. Sometimes the installers will turn on IP Whitelisting on the inbound ports, but sometimes that gets skipped.

janci 7y ago

I don't know about ICS, but CCTV cameras, HVAC controls, parking gates are too often installed by technicians with little to no IT knowledge. They ask the local IT to open port (ie. port forwarding) for remote management or servicing without knowing the implications. When offered a VPN they often refuse - they do not want or know how to use it.

noir_lord 7y ago

It’s not deliberate in the sense they want people to be vulnerable. It’s the path of least resistance in action. If you make it significantly more work to do something than not then people simply won’t do it. As the visible difference between secure and insecure is invisible to management it slides. As an industry we’ve failed all horribly at making secure the easy default option. And that’s going to haunt all of us.

bsder 7y ago
> As an industry we’ve failed all horribly at making secure the easy default option. This will never and can never happen. Security and sharing are at odds. If I don't want connectivity, I don't even buy a connected controller as it is more expensive. So, the fact that a connected controller got bought means someone wanted connectivity. Of course, the person who wanted connectivity didn't allocate any actual money to the person actually doing configuration, maintenance, auditing, updating, etc. Only at this point, do we have the fact that configuring a secured device is a pain. So, the default is to turn off all the security to get it up and running. After that, if someone can't tweak the software to turn the security back on in a couple of hours, it gets left off (remember--nobody actually allocated budget so these are unbillable, wasted hours). All of this together collides into a nice big pile of fail.
- lazulicurio 7y ago
  > > As an industry we’ve failed all horribly at making secure the easy default option. > This will never and can never happen. Security and sharing are at odds I don't think the GP was suggesting that products be made so secure to be functionally useless. There's a wide spectrum between completely open and completely locked down. But right now the gradient to go from open -> secure is so steep that most people aren't willing to put in the effort. For example, have you ever tried using OpenSSL to create a self-signed certificate? The defaults are probably not what you want. Adding on stuff like the proper certificate extensions is even more work. Products should be designed so that it's not a herculean task to add security when you're ready to take that step.
lazulicurio 7y ago

cookiecaper 7y ago

How much do engineers specializing in these systems get paid? I interviewed at a couple of places that do industrial-style systems and in one case, I got an offer that clocked in far below the requested rate (not a negotiation tactic, they declined to revise it), and at another, the recruiter couldn't hang up fast enough when she heard salary requirements. It's something like the Idiocracy Effect. In Idiocracy, the present-day world's best minds devote all their energy to eliminating hair loss and prolonging erections, eventually leading to a collapsed society. The real world is not too far off. Our best minds are dedicated to improving ad targeting and keeping users in vanity-gratifying loops on social media, while our industrial control systems that literally run the modern world wallow in the 70s. Everyone has some culpability in that. How many of us yearn to work for the local power utility (or one of their upstream vendors) rather than Facebook, Google, or the latest VC pump-and-dump scheme -- err, I mean, hot SV unicorn? Even if they did compensate competitively, are there enough hackers with a sense of [realism/duty/patriotism/INSERT_OTHER_SOCIAL_VALUE_HERE] to heed the call? There's definitely a role for some type of regulation or standards here. Industrial controls should be considered vital infrastructure that require serious and immediate investment. A brief visit to Shodan will show thousands of exposed industrial resources, and that should scare your pants off.

ChuckNorris89 7y ago
Don't know about the US market but in EU salaries in the embedded/automotive/industrial control are crap compared to frontend/app kiddies. It's definitely a tragedy of the commons, idiocracy style. Lots of embedded work related to critical infrastructure is now dumped to Eastern Europe or Asia as the good people in this field are mostly gray beards which are deemed too expensive by management.
coryrc 7y ago
I used to work on those sorts of systems. I'm now making 3x as much at FAANG. I love working on systems where you own it top-to-bottom, but not going to turn down extra piles of cash every month.
nitrogen 7y ago
How many of us yearn to work for the local power utility I actually would (or a vendor as you say) if it didn't mean earning 1/3 or less as much money. I mean, what better way to give back to the world than to apply one's hard-earned skills to improving the technologies (water, gas, power, manufacturing) that make modern society possible? It's like real life Ark or Infinifactory. But the pay is just not viable, and there are often stricter credential requirements than in valley software engineering.

jacquesgt 7y ago

I used to work in the industrial controls industry. The systems are often designed by application engineers working for the industrial control equipment’s manufacturer or distributor. In the case of distributors, the engineering work is often provided for “free” and paid for with the markup the distributor applies over their cost to purchase the components direct from the manufacturer. Those same application engineers will be involved with helping to make the sale. If a customer asks “can I connect this to the Internet?”, any response other than “of course!” is liable to result in a talking to from the sales manager for that account. Few of the incentives in that industry lead to good security. More details here: https://news.ycombinator.com/item?id=3260127 https://news.ycombinator.com/item?id=3260127

snowwindwaves 7y ago

Security is not mentioned in the specification for the control system so the company building it does nothing. When asked customer does not want to pay any more for secure remote access.

antisemiotic 7y ago

I can't say much about Switzerland, but on a slightly related note, in Poland we have automatic doors in trains that don't bother detecting if someone's in the doorway before closing. I'm honestly surprised that these didn't kill a single child yet.