7 ms·

PLC manufacturers and the controls community in general is lagging behind technologically by twenty years. It is very frustrating for a security-minded, compute

PLC manufacturers and the controls community in general is lagging behind technologically by twenty years. It is very frustrating for a security-minded, computer-literate controls engineer to work with decade old hardware and software. In a company I worked at, there were PLCs controlling sensitive processes, including heaters for welding plastic, connected to the company network via Modbus-over-TCP. A visitor with a guest access, nmap and pymodbus could read AND SET temperatures of any of the devices. A black hat could easily have started fires or leaked acid in our production floor.

WillReplyfFood 9y ago
<Rant> Worser still, they have even trouble adopting object orientation. The whole area is stagnant in a Win95 Programming kind of way. Its truly horrifying. Worser still, even the people producing the newer tools to allow for OO, do not really test these tools, so you develop in a environment where you are basically a beta tester on a budget (TC3 - looking at you). Nobody dares to move, to not break precious backwards compatability and devalue the experience of "experts" who for several generations simply did fall into every trap state-chart based design can provide. Means, every project runs into complexity walls at the end, there are no automatic proofers to check the state charts, there are no unit-tests (If i here the poor excuse -"But there is no hardware to test against" one more time). In my opinion, this industry is as ripe for disruption as it gets- if you can generate and update code from a high lvl object and process description here, and proof reliability and speed + allow for user-code - the industry would throw money at you. Half of the projects in this area are not ond budget and not on time. There is actually not even planning for that- just a factor, which usually is bigger then 2.0. Im so sick of the excuses. You cant compare conventional coding and PLC programming. Yeah, you cant, one moved forward and the other got stuck with the lowest common denominator - due to lots of non-software guys trying to get a food in. All these fb lego-players, with no clue what they do... </Rant>
- rukenshia 9y ago
  Did you ever have the pleasure of working with Siemens' SCL? I found it very frustrating to use but overall a better experience than just using the lego blocks programming you usually see. Combined with your data blocks you could at least get some structure into your programs.
  - WillReplyfFood 9y ago
    I did, i did and i used the TC synonym. The problem is- as with all state-charts fast grows in complexity to a level where nobody has a overview. Are we the first industry to encounter this. No- chip designers have this all the time, game industry has this all the time, every fucking industry using state-charts has this - all the time. So how comes, we are the only industry taking a lot of overtime all the time? <chirpin ciccadas since decadas> I have people whos VMs literally collapse under page-long state-charts. So how about breaking them up, just have small state-charts in FBs and small state charts in FBs marshalling them. Not happening. And they use assembler, not for a final tweak but as base language. And usually, when the whole mess is collapsing in on itself, like a black hole of bad design, the project manager call in some external consultants, who should be happy to work a project that is "that far along -its allmost done". A castle made from dinosaur bollocks.
    - rukenshia 9y ago
      I think the big problem in us being that slow is that you can't properly test changes. The initial programs get written under immense time pressure to put it productive and what after that? Do you want to touch your productive system and refactor it and risk causing hundreds of thousands in damages for production loss? You pretty much need to nail your initial design and can't afford to do the usual "we will take care of our technical debt later". I hear you saying "but you have debuggers/simulators!", for anyone actually having worked in the field you will know they are pretty much useless for big changes in machines that speak to hundreds of other systems, sensors, motors, etc. At least that's what my experience was and nobody has a backup factory to test changes on. In the tight schedules we had during production stops (Sunday nights, mainly) we were busy enough maintaining everything else than having fun on a PLC. I completely agree that this is all a big mess and people don't tend to write maintainable programs, they just want their machines running and this is where everyone needs to be trained and improve.
      - WillReplyfFood 9y ago
        The problem with testing changes is indeed that the environment at programming start is usually under construction and in itteration. Which is why a automatic proofer going over the state chart testing for edge cases would be great. Unittests against a simulation would allready be great if they would not test the whole process, but the working of components. Actuators and Sensors usually do not change that much- so basic test-coverage would be fine. Regarding the time, i found out there usually is time- a miserable long time at the project-end, where everyone crunches besides the allready deployed machine, personal loosing the overview of the architecture and thus leaving. We had programs where even base functionality had uncovered edge cases which would destroy parts of the machine. To detec these edge cases is absoluty doable with software or with decent planning. There is no real thinking about the life-cycle of fbs-during machine initalisation, start, stop and maintainenance. Instead something "working" is produced and then simply applied to a machine. Even companies who have some standards, meaning some internal guy who trys to keep software coherent and reusable, usually have a decay in architecture during the project. Statemachines, squished into one page long boolean statements and pages upon pages of copy paste code. Eletricians and mechanics coding with funny building blocks. This has to go.
- carlmr 9y ago
  Working in embedded systems, also with a control background and software affinity, I find we have the same mindset problems in non-manufacturing contexts as well (any kind of industry that needs embedded systems). >In my opinion, this industry is as ripe for disruption as it gets- if you can generate and update code from a high lvl object and process description here, and proof reliability and speed + allow for user-code - the industry would throw money at you. They won't, because they don't see the value in doing it right. An industry managed by mechanical engineers is not going to see how they're spending 100x as much as they need to for a worse product. Because they don't see the value in software in the first place. For them it's just this nasty thing you need to throw a lot of money at until it everything works. The mindset would need to shift to software being the main space in which problems are solved, because in 2018 it is. I guess this also turned into a </rant>
  - skgoa 9y ago
    > They won't, because they don't see the value in doing it right. An industry managed by mechanical engineers is not going to see how they're spending 100x as much as they need to for a worse product. Because they don't see the value in software in the first place. This has largely changed in the automotive industry over the last few years. We still aren't quite there yet on getting all managers to spend on security preemptively, but at least we now have had the big honchos hand down the gospel that software is important.
- taneq 9y ago
  I hear ya. The issue, though, is that the end product has to be at least readable, if not maintainable, by any half competent electrician, so lowest common denominator (ie. usually ladder logic) is as abstract as you can go. Also for most industrial controls, the system is "wide" rather than "deep", in the sense that almost everything is pretty trivial ('read inputs X and Y, set output Z if X and not Y') but there are thousands of X and Y. And finally, X etc. are tied so intimately with the hardware, which by the time the site is actually built and all the vendor packages integrated is only loosely related to the original design. So while I absolutely agree that it could be improved on significantly, the end result will still look (on the surface) quite like what we have now.
  - WillReplyfFood 9y ago
    The systems where wide, and not deep. But by now they want industry 4.0 - means they want intelligent behaving machines. As in - material flow on fail rerouting, process controlling and self-adjusting, as in self checking for upcoming flaws. All parts are tracked over network, all guides to the machine have to be electronic. And they want this with the conditions you described.
    - taneq 9y ago
      Is this in factory automation? I work in mining and everything is still very simple there, nothing more complex than a PID controller or speed ramps on some variable speed drives. I wish we could do some more advanced tuning, but the key criteria for anything we do is reliability, robustness (even in the face of sensor failures, everything is monitored and has manual overrides) and simplicity.
  - marcosdumay 9y ago
    > the end product has to be at least readable, if not maintainable, by any half competent electrician You can get that by using Python or similar languages; you can get that by programming in something like a Haskell DSL and hiding the underlining libraries; you can get that by creating an actual DSL and interpreting it on the fly. But C or C++ state machines manually encoded into structured code is one of the things that won't get you there.
- pmlnr 9y ago
  OOP isn't always needed. Most of the PCLs are very simple state machines where there really is no actual use for inheritance, or even encapsulation. It it true though that the mindset is decades behind the web-oriented world.
  - WillReplyfFood 9y ago
    Its simple at the start. And sorry, but with the stuff i work, its usually not enough. OO would allow for easier code reuse, by composing objects out of pre-existing objects, something that is usually done by copy paste today. OO would enforce encapsulation and prevent global periphery fondling by serveral free running state-charts, where its really tough to find out, who in what order did flip the switch and cause a crash in 1 of 1000 runs of machinery. List goes on... Sorry, but for my use cases the complexity is usually not completely avoidable.
  - digi_owl 9y ago
    Then again the web oriented world all too often come across as architecture astronauts...
w8rbt 9y ago
I've done this too. It's neat to watch a 'read-only' register on a PLC change because of a packet you sent it. Overflowing a buffer into an adjacent buffer remotely. It's an art that should have died long ago. PLCs need to be upgraded from C to C++ and devs should use standard C++ containers and remove all the fixed size C arrays.
- thriftwy 9y ago
  > PLCs need to be upgraded from C to C++ You are implying that C++ code is generally safer than C code, even in the hands of developers who introduce buffer overruns in the first place. This is so wrong I don't even know where to start. In C++, I doubt they will ever get it out of door. Which would solve the security problem, incidentally.
  - peoplewindow 9y ago
    Modern C++ should be safer. If you keep dynamic memory allocation to a minimum, use the STL heavily and are OK with lots of copying then it's harder to screw up because there's not much manual freeing of stuff and buffer sizes are systematically checked. Of course if you're going to leave C, why not use real time Java or some other safer language.... C++ doesn't seem like a very ambitious sort of upgrade.
    - plaguuuuuu 9y ago
      software for setting parameters for hardware systems might not require complex logic. a language with formal verification would be a good idea in that case.
  - w8rbt 9y ago
    std::vector<whatever> in C++ is safe and is nothing at all like a fixed size array of whatever in C. People who don't understand this are ignorant of the facts. They are totally different languages. C++ is much safer than C. Go ahead and try to overflow std::vector<whatever>. You can't. It's not possible. Also, it is relatively straight-forward to compile C code with a C++ compiler and adopt C++ constructs while throwing out all the old insecure C constructs. Doing this would vastly improve security and is achievable in a short period of time with little or no performance loss (it will probably run even faster). Yet, all I read about on web forums such as HN are unattainable suggestions such as re-write X in the newest, non-standard, corporate backed/controlled language that isn't nearly as tested nor as performant as ISO Standard C++. Maybe it's just me, but I find these suggestions to be totally out of touch with reality and really uniformed about the vast differences between C and C++.
cryptonector 9y ago
I mean, you're not wrong. PLCs are a digital technology evolved from... a electromechanical computers of the 50s. PLCs are not just twenty years behind. They're really just 60+ years of legacy.
- pdx 9y ago
  It's true that PLC's use ladder logic, which is a representation of relay logic from 60 years ago. There's nothing wrong with ladder logic. It's easy for electricians to understand and change. More importantly, though, it's easy to see the logic flow that makes that conveyor belt that could rip off an arm, or that 1000HP fan, or that critical pump or that critical valve, function. It's important that the safety interlocks, switches, timers, and sensor values that control that piece of equipment be all grouped together visually to help avoid the frequent surprises (bugs), so common with text based programming languages. If you want to beef up the network security to the PLC, be my guest, but don't disparage ladder logic. It's the right tool for what it's used for.
  - laythea 9y ago
    Agreed. Long live ladder logic!
    - pdx 9y ago
      It seems like there's opportunity here. A little box with raspberry pi or similar device and two CAT5 connectors to be placed inline between the PLC and the network, as well as between the network and all other network based displays, interfaces, and sensors. This would allow non-hardened equipment on a non-hardened network to be easily retrofit with encrypted connections. Edit: These already exist. [1] ;-( Oh well. Just need to have facilities start using them throughout the site. [1] https://www.lifewire.com/best-vpn-enabling-devices-4140254 https://www.lifewire.com/best-vpn-enabling-devices-4140254
      - snowwindwaves 9y ago
        Devices exist to filter out modbus data that doesn't belong [0], but I have yet to see one in the wild. Whenever I have offered plant owners additional security such as a VPN endpoint at the satellite ISPs base station, they have opted against it due to cost. [0] https://www.tofinosecurity.com/products/Tofino-Modbus-TCP-Enforcer-LSM https://www.tofinosecurity.com/products/Tofino-Modbus-TCP-En...

WillReplyfFood 9y ago

<Rant> Worser still, they have even trouble adopting object orientation. The whole area is stagnant in a Win95 Programming kind of way. Its truly horrifying. Worser still, even the people producing the newer tools to allow for OO, do not really test these tools, so you develop in a environment where you are basically a beta tester on a budget (TC3 - looking at you). Nobody dares to move, to not break precious backwards compatability and devalue the experience of "experts" who for several generations simply did fall into every trap state-chart based design can provide. Means, every project runs into complexity walls at the end, there are no automatic proofers to check the state charts, there are no unit-tests (If i here the poor excuse -"But there is no hardware to test against" one more time). In my opinion, this industry is as ripe for disruption as it gets- if you can generate and update code from a high lvl object and process description here, and proof reliability and speed + allow for user-code - the industry would throw money at you. Half of the projects in this area are not ond budget and not on time. There is actually not even planning for that- just a factor, which usually is bigger then 2.0. Im so sick of the excuses. You cant compare conventional coding and PLC programming. Yeah, you cant, one moved forward and the other got stuck with the lowest common denominator - due to lots of non-software guys trying to get a food in. All these fb lego-players, with no clue what they do... </Rant>

rukenshia 9y ago
Did you ever have the pleasure of working with Siemens' SCL? I found it very frustrating to use but overall a better experience than just using the lego blocks programming you usually see. Combined with your data blocks you could at least get some structure into your programs.
- WillReplyfFood 9y ago
  I did, i did and i used the TC synonym. The problem is- as with all state-charts fast grows in complexity to a level where nobody has a overview. Are we the first industry to encounter this. No- chip designers have this all the time, game industry has this all the time, every fucking industry using state-charts has this - all the time. So how comes, we are the only industry taking a lot of overtime all the time? <chirpin ciccadas since decadas> I have people whos VMs literally collapse under page-long state-charts. So how about breaking them up, just have small state-charts in FBs and small state charts in FBs marshalling them. Not happening. And they use assembler, not for a final tweak but as base language. And usually, when the whole mess is collapsing in on itself, like a black hole of bad design, the project manager call in some external consultants, who should be happy to work a project that is "that far along -its allmost done". A castle made from dinosaur bollocks.
  - rukenshia 9y ago
    I think the big problem in us being that slow is that you can't properly test changes. The initial programs get written under immense time pressure to put it productive and what after that? Do you want to touch your productive system and refactor it and risk causing hundreds of thousands in damages for production loss? You pretty much need to nail your initial design and can't afford to do the usual "we will take care of our technical debt later". I hear you saying "but you have debuggers/simulators!", for anyone actually having worked in the field you will know they are pretty much useless for big changes in machines that speak to hundreds of other systems, sensors, motors, etc. At least that's what my experience was and nobody has a backup factory to test changes on. In the tight schedules we had during production stops (Sunday nights, mainly) we were busy enough maintaining everything else than having fun on a PLC. I completely agree that this is all a big mess and people don't tend to write maintainable programs, they just want their machines running and this is where everyone needs to be trained and improve.
    - WillReplyfFood 9y ago
      The problem with testing changes is indeed that the environment at programming start is usually under construction and in itteration. Which is why a automatic proofer going over the state chart testing for edge cases would be great. Unittests against a simulation would allready be great if they would not test the whole process, but the working of components. Actuators and Sensors usually do not change that much- so basic test-coverage would be fine. Regarding the time, i found out there usually is time- a miserable long time at the project-end, where everyone crunches besides the allready deployed machine, personal loosing the overview of the architecture and thus leaving. We had programs where even base functionality had uncovered edge cases which would destroy parts of the machine. To detec these edge cases is absoluty doable with software or with decent planning. There is no real thinking about the life-cycle of fbs-during machine initalisation, start, stop and maintainenance. Instead something "working" is produced and then simply applied to a machine. Even companies who have some standards, meaning some internal guy who trys to keep software coherent and reusable, usually have a decay in architecture during the project. Statemachines, squished into one page long boolean statements and pages upon pages of copy paste code. Eletricians and mechanics coding with funny building blocks. This has to go.
carlmr 9y ago
Working in embedded systems, also with a control background and software affinity, I find we have the same mindset problems in non-manufacturing contexts as well (any kind of industry that needs embedded systems). >In my opinion, this industry is as ripe for disruption as it gets- if you can generate and update code from a high lvl object and process description here, and proof reliability and speed + allow for user-code - the industry would throw money at you. They won't, because they don't see the value in doing it right. An industry managed by mechanical engineers is not going to see how they're spending 100x as much as they need to for a worse product. Because they don't see the value in software in the first place. For them it's just this nasty thing you need to throw a lot of money at until it everything works. The mindset would need to shift to software being the main space in which problems are solved, because in 2018 it is. I guess this also turned into a </rant>
- skgoa 9y ago
  > They won't, because they don't see the value in doing it right. An industry managed by mechanical engineers is not going to see how they're spending 100x as much as they need to for a worse product. Because they don't see the value in software in the first place. This has largely changed in the automotive industry over the last few years. We still aren't quite there yet on getting all managers to spend on security preemptively, but at least we now have had the big honchos hand down the gospel that software is important.
taneq 9y ago
I hear ya. The issue, though, is that the end product has to be at least readable, if not maintainable, by any half competent electrician, so lowest common denominator (ie. usually ladder logic) is as abstract as you can go. Also for most industrial controls, the system is "wide" rather than "deep", in the sense that almost everything is pretty trivial ('read inputs X and Y, set output Z if X and not Y') but there are thousands of X and Y. And finally, X etc. are tied so intimately with the hardware, which by the time the site is actually built and all the vendor packages integrated is only loosely related to the original design. So while I absolutely agree that it could be improved on significantly, the end result will still look (on the surface) quite like what we have now.
- WillReplyfFood 9y ago
  The systems where wide, and not deep. But by now they want industry 4.0 - means they want intelligent behaving machines. As in - material flow on fail rerouting, process controlling and self-adjusting, as in self checking for upcoming flaws. All parts are tracked over network, all guides to the machine have to be electronic. And they want this with the conditions you described.
  - taneq 9y ago
    Is this in factory automation? I work in mining and everything is still very simple there, nothing more complex than a PID controller or speed ramps on some variable speed drives. I wish we could do some more advanced tuning, but the key criteria for anything we do is reliability, robustness (even in the face of sensor failures, everything is monitored and has manual overrides) and simplicity.
- marcosdumay 9y ago
  > the end product has to be at least readable, if not maintainable, by any half competent electrician You can get that by using Python or similar languages; you can get that by programming in something like a Haskell DSL and hiding the underlining libraries; you can get that by creating an actual DSL and interpreting it on the fly. But C or C++ state machines manually encoded into structured code is one of the things that won't get you there.
pmlnr 9y ago
OOP isn't always needed. Most of the PCLs are very simple state machines where there really is no actual use for inheritance, or even encapsulation. It it true though that the mindset is decades behind the web-oriented world.
- WillReplyfFood 9y ago
  Its simple at the start. And sorry, but with the stuff i work, its usually not enough. OO would allow for easier code reuse, by composing objects out of pre-existing objects, something that is usually done by copy paste today. OO would enforce encapsulation and prevent global periphery fondling by serveral free running state-charts, where its really tough to find out, who in what order did flip the switch and cause a crash in 1 of 1000 runs of machinery. List goes on... Sorry, but for my use cases the complexity is usually not completely avoidable.
- digi_owl 9y ago
  Then again the web oriented world all too often come across as architecture astronauts...

w8rbt 9y ago

I've done this too. It's neat to watch a 'read-only' register on a PLC change because of a packet you sent it. Overflowing a buffer into an adjacent buffer remotely. It's an art that should have died long ago. PLCs need to be upgraded from C to C++ and devs should use standard C++ containers and remove all the fixed size C arrays.

thriftwy 9y ago
> PLCs need to be upgraded from C to C++ You are implying that C++ code is generally safer than C code, even in the hands of developers who introduce buffer overruns in the first place. This is so wrong I don't even know where to start. In C++, I doubt they will ever get it out of door. Which would solve the security problem, incidentally.
- peoplewindow 9y ago
  Modern C++ should be safer. If you keep dynamic memory allocation to a minimum, use the STL heavily and are OK with lots of copying then it's harder to screw up because there's not much manual freeing of stuff and buffer sizes are systematically checked. Of course if you're going to leave C, why not use real time Java or some other safer language.... C++ doesn't seem like a very ambitious sort of upgrade.
  - plaguuuuuu 9y ago
    software for setting parameters for hardware systems might not require complex logic. a language with formal verification would be a good idea in that case.
- w8rbt 9y ago
  std::vector<whatever> in C++ is safe and is nothing at all like a fixed size array of whatever in C. People who don't understand this are ignorant of the facts. They are totally different languages. C++ is much safer than C. Go ahead and try to overflow std::vector<whatever>. You can't. It's not possible. Also, it is relatively straight-forward to compile C code with a C++ compiler and adopt C++ constructs while throwing out all the old insecure C constructs. Doing this would vastly improve security and is achievable in a short period of time with little or no performance loss (it will probably run even faster). Yet, all I read about on web forums such as HN are unattainable suggestions such as re-write X in the newest, non-standard, corporate backed/controlled language that isn't nearly as tested nor as performant as ISO Standard C++. Maybe it's just me, but I find these suggestions to be totally out of touch with reality and really uniformed about the vast differences between C and C++.

cryptonector 9y ago

I mean, you're not wrong. PLCs are a digital technology evolved from... a electromechanical computers of the 50s. PLCs are not just twenty years behind. They're really just 60+ years of legacy.

pdx 9y ago
It's true that PLC's use ladder logic, which is a representation of relay logic from 60 years ago. There's nothing wrong with ladder logic. It's easy for electricians to understand and change. More importantly, though, it's easy to see the logic flow that makes that conveyor belt that could rip off an arm, or that 1000HP fan, or that critical pump or that critical valve, function. It's important that the safety interlocks, switches, timers, and sensor values that control that piece of equipment be all grouped together visually to help avoid the frequent surprises (bugs), so common with text based programming languages. If you want to beef up the network security to the PLC, be my guest, but don't disparage ladder logic. It's the right tool for what it's used for.
- laythea 9y ago
  Agreed. Long live ladder logic!
  - pdx 9y ago
    It seems like there's opportunity here. A little box with raspberry pi or similar device and two CAT5 connectors to be placed inline between the PLC and the network, as well as between the network and all other network based displays, interfaces, and sensors. This would allow non-hardened equipment on a non-hardened network to be easily retrofit with encrypted connections. Edit: These already exist. [1] ;-( Oh well. Just need to have facilities start using them throughout the site. [1] https://www.lifewire.com/best-vpn-enabling-devices-4140254 https://www.lifewire.com/best-vpn-enabling-devices-4140254
    - snowwindwaves 9y ago
      Devices exist to filter out modbus data that doesn't belong [0], but I have yet to see one in the wild. Whenever I have offered plant owners additional security such as a VPN endpoint at the satellite ISPs base station, they have opted against it due to cost. [0] https://www.tofinosecurity.com/products/Tofino-Modbus-TCP-Enforcer-LSM https://www.tofinosecurity.com/products/Tofino-Modbus-TCP-En...