5 ms·

Another odd observation about the industry. For better or for worse, I inherited a little system controlled by a Fanuc PLC. It had a bug and would occasionally

by amluto 2mo ago

Another odd observation about the industry. For better or for worse, I inherited a little system controlled by a Fanuc PLC. It had a bug and would occasionally get stuck in a bad state. By some minor miracle, I managed to track down the person who had originally programmed it (now retired and moved out of state), and he emailed me a file and introduced me to his apprentice (who lives in a third state). I found a trial version of the design software, opened the file and found the bug (despite never having seen this style of programming before). Since I had no way to upload a new file (and since it wasn’t clear that uploading it without a matching outdated and not easily available copy of the programming environment would be wise), I convinced the apprentice to come visit while we was nearby for another client. He did not see the bug even when I pointed it out, but I convinced him to make the change and upload the modified file. Now the system works. (He had a Windows laptop with a giant folder of customer files, of course.) While he was on site, I asked him why there was nothing on the PLC or its enclosure to identify his old boss or give any contact information, and he seemed surprised and told me that no one ever does that. I asked how a new owner is supposed to get support, and he shrugged. If this thing ever fails, I’ll probably replace it with an Arduino or an ESP32 or something along those lines. Or I’ll just find something off the shelf to replace the entire system.

polishdude20 2mo ago
The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it.
- deleted 2mo ago
  [deleted]
- ssl-3 2mo ago
  You mean like, just hang a thumb drive on a lanyard inside of the cabinet -- with source code and such? That will probably just disappear. Maybe with good intention or maybe with bad, but it's probably gone either way. If it doesn't disappear, then it's hanging right there for any of the competitors to use. That's a problem for the original installer's ongoing employment. And if it includes the programming software, then that lets Joe (from over in shipping) have a go at rejiggering the packaging machine. That's a problem for whoever has to pay someone with a clue to show up and fix it.
  - skinfaxi 2mo ago
    > If it doesn't disappear, then it's hanging right there for any of the competitors to use. That's a problem for the original installer's ongoing employment. Is that really an issue? Do PLC vendors have concerns about other vendors turning up and yanking USB drives out of critical infrastructure with reckless abandon? Do the facilities themselves see no issue with potential vendors yanking operational utilities out of their systems while they are running?
    - ssl-3 2mo ago
      Eh? It's just a thumb drive, on a lanyard, hanging in a cabinet. It's not like it has guards posted. :) Maybe Joe (from shipping) took it back to his desk to have a squiz at it and try to passively learn something new during some otherwise-downtime and never returned it -- years ago. That cabinet is in his area, so he has the key. Maybe Tim (the plant electrician) borrowed it to get something else done, and it unintentionally disappeared somewhere along the way. Tim has all of the keys for all of the things. Maybe Emily (the ubergeek who prides herself on being able to code her way out of a mess on any system) had it in her office before she got hit by a car and lost the ability to communicate with polysyllabic words. She borrowed Joe's cabinet key after he mentioned an issue. Either way, it was just a thumb drive that had been inside of the cabinet, and now it is gone.
      - skinfaxi 2mo ago
        Things like thumb drives physically connected to cabinets can just disappear in your environment with no followup whatsoever? And that's regular, expected, and desired?
        ssl-3 2mo ago
        So they conduct a formal investigation. Top Men descend upon the facility to figure out whether we blame Joe, or Tim, or Emily. Heads roll. And the thumb drive is still gone, isn't it? --- No, it's not regular. It's not desired. Does shit never go wrong in your world? If not, then: Perhaps you should start expecting it to. :)
- coryrc 2mo ago
  Relatedly, I made an embedded product. I built a live CD (yes this was a while ago) for my employer with the entire development environment. As long as you can find an x86 machine with CD-ROM you could build, compile, test, and program using the exact environment I did twenty years ago. Maybe today I'd do micropython on ESP32. Download text file from device, edit, upload back on.
  - aliasxneo 2mo ago
    There's at least one customer out there who has an RPi with NixOS on it from me. Probably still works to this day. It was the best use of NixOS I think I've ever found.
    - Kim_Bruning 2mo ago
      I've switched up to NUC-ish hardware which WAS cheap until recently. All the components just a bit higher quality than the lowest-bidder you get on the Pi. Run nixos on that and you've got a really solid platform.
    - rurban 2mo ago
      More probably worst use of NixOS. In embedded we care for every single kb, and don't have tolerance for this kind of space waste management system
      - voakbasda 2mo ago
        Embedded means different things. Personally I have a hard time calling anything that runs Linux as embedded, having cut my teeth on microcontrollers.
        hiAndrewQuinn 2mo ago
        If it's a custom Yocto thing or it involved buildroot at any state of provisioning it gets the moniker "quasi-embedded" to me. Doubly so if it never actually connects to the internet, unlike these water controllers, apparently.
- hommelix 2mo ago
  > The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it. This is already the case. Some PLC allows you to store and retrieve the source code of the programmed directly in the flash of the PLC. Sometimes this feature is behind a higher access level or password.
  - NotMichaelBay 2mo ago
    Is it version controlled too?
- KaiserPro 2mo ago
  Nooooope nope nope. An entire class of controller with really shit security now (as in scanning a network with PLC controllers has a non-trivial chance of rebooting or freezing any number of devices on the network) trained to just plug USB sticks in? I understand the idea, and its laudable. But the entire model of PLC security needs root and branches overhaul.
spauldo 2mo ago
A microcontroller based solution is rarely a good replacement for a PLC. You can find integrators willing to come out and troubleshoot your device in the field a lot more easily than embedded progammers and vendor support is usually excellent. When your plant is down, that sort of thing matters. PLCs handle harsh environments for decades, whereas with a microcontroller all the thermal issues and power conditioning are all on you. When something fails, it's usually just a matter of pulling a spare module off the shelf and plugging it in. If that's an old GE 90-30 (a common FANUC system), it's time to replace it. You're currently in that period when the CPU has reached end of life but the modules are still supported and the vendor has an easy upgrade path. Emerson owns the brand now, and can sell you a new unit that'll run your old program with minimal changes. You can call them and get a list of integrators in your area. It'll be pricey, but you won't need to do it again for another quarter century.
- giantg2 2mo ago
  "but you won't need to do it again for another quarter century." Hmm, so how do they handle the new cyber threats, since most of these are connected? It seems upgrades would need to be performed to address new threats. And just because something is not internet connected doesn't make it immune (eg stuxnet).
  - black6 2mo ago
    They shouldn't be connected is the point of the article. All PLC and DCS manufacturers are constantly working on threat mitigation and pushing out updates to address vulnerabilities. It's on the Process Controls team to decide what is an actual threat that requires patching.
    - giantg2 2mo ago
      Iran's PLC weren't connected. How did that turn out?
      - spauldo 2mo ago

by amluto 2mo ago

polishdude20 2mo ago

The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it.

deleted 2mo ago
[deleted]
ssl-3 2mo ago
You mean like, just hang a thumb drive on a lanyard inside of the cabinet -- with source code and such? That will probably just disappear. Maybe with good intention or maybe with bad, but it's probably gone either way. If it doesn't disappear, then it's hanging right there for any of the competitors to use. That's a problem for the original installer's ongoing employment. And if it includes the programming software, then that lets Joe (from over in shipping) have a go at rejiggering the packaging machine. That's a problem for whoever has to pay someone with a clue to show up and fix it.
- skinfaxi 2mo ago
  > If it doesn't disappear, then it's hanging right there for any of the competitors to use. That's a problem for the original installer's ongoing employment. Is that really an issue? Do PLC vendors have concerns about other vendors turning up and yanking USB drives out of critical infrastructure with reckless abandon? Do the facilities themselves see no issue with potential vendors yanking operational utilities out of their systems while they are running?
  - ssl-3 2mo ago
    Eh? It's just a thumb drive, on a lanyard, hanging in a cabinet. It's not like it has guards posted. :) Maybe Joe (from shipping) took it back to his desk to have a squiz at it and try to passively learn something new during some otherwise-downtime and never returned it -- years ago. That cabinet is in his area, so he has the key. Maybe Tim (the plant electrician) borrowed it to get something else done, and it unintentionally disappeared somewhere along the way. Tim has all of the keys for all of the things. Maybe Emily (the ubergeek who prides herself on being able to code her way out of a mess on any system) had it in her office before she got hit by a car and lost the ability to communicate with polysyllabic words. She borrowed Joe's cabinet key after he mentioned an issue. Either way, it was just a thumb drive that had been inside of the cabinet, and now it is gone.
    - skinfaxi 2mo ago
      Things like thumb drives physically connected to cabinets can just disappear in your environment with no followup whatsoever? And that's regular, expected, and desired?
      - ssl-3 2mo ago
        So they conduct a formal investigation. Top Men descend upon the facility to figure out whether we blame Joe, or Tim, or Emily. Heads roll. And the thumb drive is still gone, isn't it? --- No, it's not regular. It's not desired. Does shit never go wrong in your world? If not, then: Perhaps you should start expecting it to. :)
coryrc 2mo ago
Relatedly, I made an embedded product. I built a live CD (yes this was a while ago) for my employer with the entire development environment. As long as you can find an x86 machine with CD-ROM you could build, compile, test, and program using the exact environment I did twenty years ago. Maybe today I'd do micropython on ESP32. Download text file from device, edit, upload back on.
- aliasxneo 2mo ago
  There's at least one customer out there who has an RPi with NixOS on it from me. Probably still works to this day. It was the best use of NixOS I think I've ever found.
  - Kim_Bruning 2mo ago
    I've switched up to NUC-ish hardware which WAS cheap until recently. All the components just a bit higher quality than the lowest-bidder you get on the Pi. Run nixos on that and you've got a really solid platform.
  - rurban 2mo ago
    More probably worst use of NixOS. In embedded we care for every single kb, and don't have tolerance for this kind of space waste management system
    - voakbasda 2mo ago
      Embedded means different things. Personally I have a hard time calling anything that runs Linux as embedded, having cut my teeth on microcontrollers.
      - hiAndrewQuinn 2mo ago
        If it's a custom Yocto thing or it involved buildroot at any state of provisioning it gets the moniker "quasi-embedded" to me. Doubly so if it never actually connects to the internet, unlike these water controllers, apparently.
hommelix 2mo ago
> The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it. This is already the case. Some PLC allows you to store and retrieve the source code of the programmed directly in the flash of the PLC. Sometimes this feature is behind a higher access level or password.
- NotMichaelBay 2mo ago
  Is it version controlled too?
KaiserPro 2mo ago
Nooooope nope nope. An entire class of controller with really shit security now (as in scanning a network with PLC controllers has a non-trivial chance of rebooting or freezing any number of devices on the network) trained to just plug USB sticks in? I understand the idea, and its laudable. But the entire model of PLC security needs root and branches overhaul.

spauldo 2mo ago

A microcontroller based solution is rarely a good replacement for a PLC. You can find integrators willing to come out and troubleshoot your device in the field a lot more easily than embedded progammers and vendor support is usually excellent. When your plant is down, that sort of thing matters. PLCs handle harsh environments for decades, whereas with a microcontroller all the thermal issues and power conditioning are all on you. When something fails, it's usually just a matter of pulling a spare module off the shelf and plugging it in. If that's an old GE 90-30 (a common FANUC system), it's time to replace it. You're currently in that period when the CPU has reached end of life but the modules are still supported and the vendor has an easy upgrade path. Emerson owns the brand now, and can sell you a new unit that'll run your old program with minimal changes. You can call them and get a list of integrators in your area. It'll be pricey, but you won't need to do it again for another quarter century.

giantg2 2mo ago
"but you won't need to do it again for another quarter century." Hmm, so how do they handle the new cyber threats, since most of these are connected? It seems upgrades would need to be performed to address new threats. And just because something is not internet connected doesn't make it immune (eg stuxnet).
- black6 2mo ago
  They shouldn't be connected is the point of the article. All PLC and DCS manufacturers are constantly working on threat mitigation and pushing out updates to address vulnerabilities. It's on the Process Controls team to decide what is an actual threat that requires patching.
  - giantg2 2mo ago
    Iran's PLC weren't connected. How did that turn out?
    - spauldo 2mo ago