4 ms·

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data cen

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worked with an integrator who was working on an AHU feeding an extremely critical portion of a datacenter (I was a lead by this point and mostly played babysitter). During certain points of the day you couldn't open the door to this room due to negative pressure because the logic was over-ramping the exhaust fans. As I watched this contractor work, I saw him open his laptop, with Windows on it (because Microsoft has had a death grip on this industry for decades now), and proceed to backup the PLC program into a massive folder with God knows how many other "customer projects" he was carrying around in this thing. He then proceeded to go do some physical checks in the field, came back, and prepared to upload the fixed program. As I watched, I noticed he _grabbed a backup from ANOTHER customer_ and I immediately had to intervene. Who knows what untold damage I saved from that single move. I tell this story to demonstrate just how far into the dark ages this industry is. I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID and why they might want to consider a disaster recovery plan for their multi-million dollar mechanical plant. This industry is in _desperate_ need of strong technical help, but unfortunately the "higher ups" tend to be the same people who are "comfortable" with the way thing are and refuse to move. I literally tried for a decade before giving up and moving into software engineering proper. So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure.

elevation 2mo ago
A colleague of mine transitioned from a senior IC role to manager of a PLC group in the same company. Their methods were just as unsophisticated as you describe, and now he is attempting to evangelize/impose the finer parts of software engineering discipline, such as source control and integration tests (which can be trickier when it requires a hardware test bed.) Bringing some new tools to bear can be a fun job, as long as the team trusts you.
- aliasxneo 2mo ago
  Yes, the fortune 50 company I spoke of was running some of the most advanced data centers in the world. There was literally a team of world class IT specialists walking the same corridors as the industrial automation team but they might as well have spoken different languages. I remember a time where I was beating the drums on security and ended up in a meeting with a senior red team member in the company. This person was absolutely convinced we were not running Windows Server 2008 anywhere in the company (the year was 2019 at the time of that meeting). Needless to say, he was very concerned when I showed him the 50+ servers running it globally, all covering critical infrastructure. I think eventually Ragnarok will happen and things will improve. I just hope it's not as detrimental as it seems setup to be.
- AlotOfReading 2mo ago
  integration tests (which can be trickier when it requires a hardware test bed.) I use this as a fizzbuzz-type test when I'm interviewing at hardware companies: do they have development hardware in a rack with programmable power supplies and mini-PCs (or similar)? It's a low, low bar for testing, and rules surprisingly many companies. They'll often just have The Guy running manual tests instead.
  - voakbasda 2mo ago
    Wow. I’d kill for even a guy running my hardware tests. Most of my jobs barely seem to leave time to write the code, before it needs to be scraped off my monitor and poured into the customer’s systems. The ones like you describe with actual hardware labs are a dream when contrasted to those nightmares.
    - Karrot_Kream 2mo ago
      Does management sell unrealistic deadlines to the vendors that buy your solutions? Are you bound by impossible support contracts? Did the firm just let go of a lot of people and expect everyone to finish work quickly? Curious how the economic/management pressures here work as I abandoned hardware very early in my career (from an internship) for similar reasons.
Kim_Bruning 2mo ago
Oh I can tell stories too! Meanwhile these guys often have huge amounts of practical experience as electricians or control technicians/engineers to go with their PLC skills, so you can't just say we should get rid of them either. Breaking into the industrial market is tricky if you don't have connections too. And if you're hired as the PLC programmer, it's sometimes an afterthought AFTER the plant is already built. "What do you mean it'll take another month? The plant is finished, isn't it?". Oh, and some projects ban "PC"s to begin with. Which sort of excludes any kind of PC programmer. And it sort of even makes sense. A lot of default PC behaviors (especially commercial software), are no longer user-unfriendly but potentially very expensive or even user-lethal when attached to a physical plant. Sounds like I could learn some things from you (and maybe vice versa). Poke me on the email in my HN profile!
- aliasxneo 2mo ago
  Yes, I don't mean to poke fun solely on the integrators. I've seen the "software first" type land in the same role and flail just as hard, if not harder. I recall one individual, who was quite proud of his "beautiful" ladder logic code, after about 3 months of being at the company ended up destroying a $100k+ chiller plant by cavitating all of the pumps because they didn't understand head pressure. I think that's one of the core difficulties with PLC programming. You have to have strong knowledge on traditional science fields like thermal dynamics, material sciences, fluid mechanics, etc., while also understanding the limitations of a 16 bit floating point integer and why overflowing that can be catastrophic.
  - amluto 2mo ago
    Sometimes I wonder whether one could design PLC-style software by first expressing the system’s constraints and then writing the actual code and having either (preferably) formal verification that the code will not violate the constraints or (less preferably) enforced assertions that will send the machine into a safe state if the constraints do not hold. The system I mentioned in my other post controlled some VFDs, and one would occasionally get stuck running at minimum speed forever instead of turning all the off when it should have. Fortunately the only harm done was a stupid waste of power and and nothing was physically damaged. If it had gotten stuck at maximum speed it might have been a different story.
TacticalCoder 2mo ago
> So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure. Same with SCADA: just as bad as what you describe.
- aliasxneo 2mo ago
  We once reversed engineered Schneider Electric's ION protocol because they wanted to charge us $10k per "seat" to get software that could interact with it directly. We took home a controller and developed a Python program that could intercept and understand the RS-232 comms. We then developed our own system to interact with it. It was all unencrypted. I just did a quick search and it looks like "Secure ION" is now a thing, introduced in the year 2023. Which just proves my point.
  - voakbasda 2mo ago
    Some of Schneider Electric’s software is a house of cards. Source: I maintained some systems for them a few years back. Never will they ever get my money again for anything. To wit, I miss the days when APC was a brand that I could trust.
technion 2mo ago
Im supposed to be doing endpoint work with people working in this field amd basically have to convince compliance they'll need to be local administrators to do their job. They get onsite and dont know what app they'll need until they look at a plc model. They'll download something from a .tw ftp server and its an unsigned executable that absolutely must run as admin to program these things. It is amazing how much of the sysadmin community just doesnt believe this is a thing you need to work with, everyone insisting its just security people being lazy and so on.
- tedd4u 2mo ago
  Oof, that's bad. Iran got burned real bad by Stuxnet, you know they would love to score big with PLC sabotage.
  - hackernud3s 2mo ago
    I feel like this is, worst case scenario, people drinking bottled water for a week. It's no stuxnet.
    - tzs 2mo ago
      If a municipal water system was down wouldn't it affect all water to residences, not just drinking water? Would it be feasible for people to flush their toilets with bottled water for a week?
      - hackernud3s 2mo ago
        Pooping in the woods or driving to a rest stop for a week might seem like a stuxnet level event, but it's really not.
steve_adams_86 2mo ago
Everything you're describing mirrors my experience with remote scientific deployments and lab equipment (wet labs in particular). I support a team which manages dozens of these, and while it's shocking how inefficient and unsafe their practices are, it's similarly shocking how poorly the industry has kept up with or even tried to establish better practices. We get brand new equipment that costs in the realm of $10k for a tiny logger, and the thing is barely capable of remotely managed deployment of version controlled firmware. There are no tools in place to allow proper testing of the firmware unless you have the cash to buy extras and build test harnesses from them. The firmware itself is often in languages that don't have proper testing frameworks, type systems, IDEs, etc. It's HARD to do the right thing. Dragging and dropping files into proprietary hardware management programs is the de facto standard. Then you layer on top the unfortunate reality that sometimes electricity does weird stuff, people design weird circuits or wire the wrong components in, and firmware tends to have to deal with non-deterministic inputs a lot more often than, say, an API on the web. It's rough. The pay is also so much worse in my experience.
- b40d-48b2-979e 2mo ago
  Not only is the pay worse (like literally half of what you make doing webdev for a corp), but they mandate you be on site at a factory working first shift like you're on the production floor or something, clocking in and out when you're a salary worker. It's an awful culture.
- analog31 2mo ago
  >>> The pay is also so much worse in my experience. The result is that anybody who can program well enough to develop software is doing so. Those with any programming skills who remain behind are smart-but-undisciplined programmers, people who have a reason to be in a particular geographic area, too old to think about changing careers, or just plain nuts. This is what "lack of investment in manufacturing, research, and infrastructure" looks like. Sow what you reap. Disclosure: I might be one of those people. I came out of grad school in physics research, and programmed an entire plant. Thankfully, that was before it was economical to put every controller on the Internet.
tamimio 2mo ago
The worst part -as I mentioned in another comment few days earlier- is most of the network is ancient, dialup, poor architecture designed by people who barely touched a computer, very old OSes, one utility had windows 3.1 a couple years ago! For them as long as it’s working, don’t touch it, because a downtime is far more costly and it will bring many eyes and attention to them than just sit and hope nothing will happen. Physical security is a major gap too, I have been to many locations where the field server panels are just behind the door, the RTUs are just closed with panel keys, even the whole facility are using some old HID cards that you can unlock in few seconds, this is even in R&D that beyond automation but also in robotic ones.
amluto 2mo ago
Another odd observation about the industry. For better or for worse, I inherited a little system controlled by a Fanuc PLC. It had a bug and would occasionally get stuck in a bad state. By some minor miracle, I managed to track down the person who had originally programmed it (now retired and moved out of state), and he emailed me a file and introduced me to his apprentice (who lives in a third state). I found a trial version of the design software, opened the file and found the bug (despite never having seen this style of programming before). Since I had no way to upload a new file (and since it wasn’t clear that uploading it without a matching outdated and not easily available copy of the programming environment would be wise), I convinced the apprentice to come visit while we was nearby for another client. He did not see the bug even when I pointed it out, but I convinced him to make the change and upload the modified file. Now the system works. (He had a Windows laptop with a giant folder of customer files, of course.) While he was on site, I asked him why there was nothing on the PLC or its enclosure to identify his old boss or give any contact information, and he seemed surprised and told me that no one ever does that. I asked how a new owner is supposed to get support, and he shrugged. If this thing ever fails, I’ll probably replace it with an Arduino or an ESP32 or something along those lines. Or I’ll just find something off the shelf to replace the entire system.
- polishdude20 2mo ago
  The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it.
  - deleted 2mo ago
    [deleted]
  - ssl-3 2mo ago
    You mean like, just hang a thumb drive on a lanyard inside of the cabinet -- with source code and such? That will probably just disappear. Maybe with good intention or maybe with bad, but it's probably gone either way. If it doesn't disappear, then it's hanging right there for any of the competitors to use. That's a problem for the original installer's ongoing employment. And if it includes the programming software, then that lets Joe (from over in shipping) have a go at rejiggering the packaging machine. That's a problem for whoever has to pay someone with a clue to show up and fix it.
procarch2019 2mo ago
Sounds like we have a similar development path. People fail to understand how complex some of these systems can be, especially since you have to have the project to understand what registers/variables are what. Some when Joe the contractor does a job and walks away, you better make sure they give you the project and you store it in a nice safe place. A good system integrator is worth their weight in gold. Sure they cost more, but getting a whole package turned over to you is worth a million more than 5 years into the lifecycle of a factory when someone wants to make mods/fix a bug/etc and has to reinvent the whole car, not just the wheel. This industry is always 10-20 years in the past. My company’s preferred vendor only just started supporting virtualization (this is for a DCS) in the past 10 years. I still have to tell my sales people to provide A/V and minimalistic backup and recovery on every project (they essentially cost nothing compared to the rest of any project).
- closeparen 2mo ago
  I enjoy watching Cursed Controls on YouTube. For him, the gold standard is a flash drive with the program on it physically hanging out in the control cabinet.
xtajv 2mo ago
Storytime: Once upon a time, I was a young grasshopper at a small consulting firm at which every employee was issued a Windows machine, our source control system was "\Customers" on C: drive, and our deployment strategy was "putty + copy-paste + hot-reload". One morning, I heard something terrible while my boss's boss's machine was booting. Something akin to grinding. The thing was angry. I gave boss^2 a heads-up that his hard drive might be failing, and that he might want to run a S.M.A.R.T. check on the poor thing. I also asked where the backup hard drives were, because I was brand-new and assumed that I just hadn't been issued one yet. I checked the supply closet, found no hard drives, popped over to the office admin person, and recommended a deal I'd seen on some WD black drives before I realized that the place had gone quiet. Everyone looked at me like I was from Mars. Exactly 7 days later, boss^2's hard drive failed. We lost a week of work and had to zero out a 5 days x 3 employees worth of billable hours. We also ended up delivering late. The client was pissed. Based on the nasty looks that I got afterwards, it appeared that the standard assumption was that I had tampered with the drive to prove a point. (Um, nope). I have since learned to ask prospective employers about their backup strategy.
- MajorTakeaway 2mo ago
  Nasty looks seems to be the sign of them realizing how you proved them to be incompetent. Incompetence will cost them their job. If I had fuck you money I'd just call them incompetent fucks and laugh at them.
  - deleted 2mo ago
    [deleted]
bitwize 2mo ago
> This industry is in _desperate_ need of strong technical help, but unfortunately the "higher ups" tend to be the same people who are "comfortable" with the way thing are and refuse to move. I literally tried for a decade before giving up and moving into software engineering proper. As soon as AI can automate the development and deployment of physical infrastructure and the firmware that drives it, watch how willing to embrace the "cutting edge" those same higher-ups become.
jdw64 2mo ago
I agree. The problem is this: most companies using PLCs work in isolated networks with internet disconnected, based on confidentiality. There are many difficulties even with using git, because the structure forces you to implement features in a way that makes it hard to separate them. Branching, in particular, doesn't work well with PLC ladder logic—PLC uses proprietary binary formats that make diff tracking impossible, so you're forced into a unified development approach
PunchyHamster 2mo ago
I mean it worked fine for 50 years, surely it will work fine for another 50
tda 2mo ago
Yep, everything is lacking. No version control, ladder diagrams instead of written code, goto statements as primary control flow in code. A $500 PLC with less processing power than a 1985 086 and 1 whole MB of memory. Upgradable with a $200 license to 1.5MB, only programmable with a 20GB monstrosity of a piece of software in which download means copying a program.e to the PLC, and upload means downloading a backup to your laptop. That world needs some disruption really really bad
- mikestew 2mo ago
  Hey, hey, a little trigger warning over here? My last job before retirement was in that world. If it had been my first software job, it would have been my last.
KellyCriterion 2mo ago
May I ask: In which year was this? > I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID < The thing is: Big companies, even non critical are usually bonded to some frameworks/regulatory standards/etc. So, mentioning that they didnt know NAS or even RAID (which is very old, end of 80s), makes me wondering in which decade this happened?
- structural 2mo ago
  I've seen similar things in this decade. Lots of stuff still running where the last change in design was 20+ years ago, the current team knows how to keep the lights on and replace parts, and calls a vendor if something goes end of life and asks for a replacement part, then configures it the same way that the older one was. It's quite reliable, and does not require paying for a team of people who wants to try new things. Once this happens for a long while there's no one in the building left who's going to advocate for change, since the only thing all that fancy new technology brings is the next 5 years of nobody getting a bonus because operational metrics weren't being met anymore. You have to try really hard to convince these people that the thing that they have seen on a daily basis to work just fine for the past twenty years is no longer good enough, because all their evidence says that it is. And you should also listen to them much more than people who just want to use the latest thing typically do, because they do have the decades of experience seeing how things actually fail.
BrandoElFollito 2mo ago
I was on the cybersecurity side of that, "advising" huge, multinational companies that hold the world together. Oh boy. I was told for instance once that my concerns about an open, unencrypted Wi-Fi for SCADA systems were not a big deal because "we are in the desert anyway, who would come here". I saw terrible things and people who were made "cybersecurity expert" when they did not want to. They were sent to courses which were totally useless and the architecture was completely fucked up anyway. I often wondered over the years why we have so little significant incidents in the utilities sector.

elevation 2mo ago

A colleague of mine transitioned from a senior IC role to manager of a PLC group in the same company. Their methods were just as unsophisticated as you describe, and now he is attempting to evangelize/impose the finer parts of software engineering discipline, such as source control and integration tests (which can be trickier when it requires a hardware test bed.) Bringing some new tools to bear can be a fun job, as long as the team trusts you.

aliasxneo 2mo ago
Yes, the fortune 50 company I spoke of was running some of the most advanced data centers in the world. There was literally a team of world class IT specialists walking the same corridors as the industrial automation team but they might as well have spoken different languages. I remember a time where I was beating the drums on security and ended up in a meeting with a senior red team member in the company. This person was absolutely convinced we were not running Windows Server 2008 anywhere in the company (the year was 2019 at the time of that meeting). Needless to say, he was very concerned when I showed him the 50+ servers running it globally, all covering critical infrastructure. I think eventually Ragnarok will happen and things will improve. I just hope it's not as detrimental as it seems setup to be.
AlotOfReading 2mo ago
integration tests (which can be trickier when it requires a hardware test bed.) I use this as a fizzbuzz-type test when I'm interviewing at hardware companies: do they have development hardware in a rack with programmable power supplies and mini-PCs (or similar)? It's a low, low bar for testing, and rules surprisingly many companies. They'll often just have The Guy running manual tests instead.
- voakbasda 2mo ago
  Wow. I’d kill for even a guy running my hardware tests. Most of my jobs barely seem to leave time to write the code, before it needs to be scraped off my monitor and poured into the customer’s systems. The ones like you describe with actual hardware labs are a dream when contrasted to those nightmares.
  - Karrot_Kream 2mo ago
    Does management sell unrealistic deadlines to the vendors that buy your solutions? Are you bound by impossible support contracts? Did the firm just let go of a lot of people and expect everyone to finish work quickly? Curious how the economic/management pressures here work as I abandoned hardware very early in my career (from an internship) for similar reasons.

Kim_Bruning 2mo ago

Oh I can tell stories too! Meanwhile these guys often have huge amounts of practical experience as electricians or control technicians/engineers to go with their PLC skills, so you can't just say we should get rid of them either. Breaking into the industrial market is tricky if you don't have connections too. And if you're hired as the PLC programmer, it's sometimes an afterthought AFTER the plant is already built. "What do you mean it'll take another month? The plant is finished, isn't it?". Oh, and some projects ban "PC"s to begin with. Which sort of excludes any kind of PC programmer. And it sort of even makes sense. A lot of default PC behaviors (especially commercial software), are no longer user-unfriendly but potentially very expensive or even user-lethal when attached to a physical plant. Sounds like I could learn some things from you (and maybe vice versa). Poke me on the email in my HN profile!

aliasxneo 2mo ago
Yes, I don't mean to poke fun solely on the integrators. I've seen the "software first" type land in the same role and flail just as hard, if not harder. I recall one individual, who was quite proud of his "beautiful" ladder logic code, after about 3 months of being at the company ended up destroying a $100k+ chiller plant by cavitating all of the pumps because they didn't understand head pressure. I think that's one of the core difficulties with PLC programming. You have to have strong knowledge on traditional science fields like thermal dynamics, material sciences, fluid mechanics, etc., while also understanding the limitations of a 16 bit floating point integer and why overflowing that can be catastrophic.
- amluto 2mo ago
  Sometimes I wonder whether one could design PLC-style software by first expressing the system’s constraints and then writing the actual code and having either (preferably) formal verification that the code will not violate the constraints or (less preferably) enforced assertions that will send the machine into a safe state if the constraints do not hold. The system I mentioned in my other post controlled some VFDs, and one would occasionally get stuck running at minimum speed forever instead of turning all the off when it should have. Fortunately the only harm done was a stupid waste of power and and nothing was physically damaged. If it had gotten stuck at maximum speed it might have been a different story.

TacticalCoder 2mo ago

> So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure. Same with SCADA: just as bad as what you describe.

aliasxneo 2mo ago
We once reversed engineered Schneider Electric's ION protocol because they wanted to charge us $10k per "seat" to get software that could interact with it directly. We took home a controller and developed a Python program that could intercept and understand the RS-232 comms. We then developed our own system to interact with it. It was all unencrypted. I just did a quick search and it looks like "Secure ION" is now a thing, introduced in the year 2023. Which just proves my point.
- voakbasda 2mo ago
  Some of Schneider Electric’s software is a house of cards. Source: I maintained some systems for them a few years back. Never will they ever get my money again for anything. To wit, I miss the days when APC was a brand that I could trust.

technion 2mo ago

Im supposed to be doing endpoint work with people working in this field amd basically have to convince compliance they'll need to be local administrators to do their job. They get onsite and dont know what app they'll need until they look at a plc model. They'll download something from a .tw ftp server and its an unsigned executable that absolutely must run as admin to program these things. It is amazing how much of the sysadmin community just doesnt believe this is a thing you need to work with, everyone insisting its just security people being lazy and so on.

tedd4u 2mo ago
Oof, that's bad. Iran got burned real bad by Stuxnet, you know they would love to score big with PLC sabotage.
- hackernud3s 2mo ago
  I feel like this is, worst case scenario, people drinking bottled water for a week. It's no stuxnet.
  - tzs 2mo ago
    If a municipal water system was down wouldn't it affect all water to residences, not just drinking water? Would it be feasible for people to flush their toilets with bottled water for a week?
    - hackernud3s 2mo ago
      Pooping in the woods or driving to a rest stop for a week might seem like a stuxnet level event, but it's really not.

steve_adams_86 2mo ago

Everything you're describing mirrors my experience with remote scientific deployments and lab equipment (wet labs in particular). I support a team which manages dozens of these, and while it's shocking how inefficient and unsafe their practices are, it's similarly shocking how poorly the industry has kept up with or even tried to establish better practices. We get brand new equipment that costs in the realm of $10k for a tiny logger, and the thing is barely capable of remotely managed deployment of version controlled firmware. There are no tools in place to allow proper testing of the firmware unless you have the cash to buy extras and build test harnesses from them. The firmware itself is often in languages that don't have proper testing frameworks, type systems, IDEs, etc. It's HARD to do the right thing. Dragging and dropping files into proprietary hardware management programs is the de facto standard. Then you layer on top the unfortunate reality that sometimes electricity does weird stuff, people design weird circuits or wire the wrong components in, and firmware tends to have to deal with non-deterministic inputs a lot more often than, say, an API on the web. It's rough. The pay is also so much worse in my experience.

b40d-48b2-979e 2mo ago
Not only is the pay worse (like literally half of what you make doing webdev for a corp), but they mandate you be on site at a factory working first shift like you're on the production floor or something, clocking in and out when you're a salary worker. It's an awful culture.
analog31 2mo ago
>>> The pay is also so much worse in my experience. The result is that anybody who can program well enough to develop software is doing so. Those with any programming skills who remain behind are smart-but-undisciplined programmers, people who have a reason to be in a particular geographic area, too old to think about changing careers, or just plain nuts. This is what "lack of investment in manufacturing, research, and infrastructure" looks like. Sow what you reap. Disclosure: I might be one of those people. I came out of grad school in physics research, and programmed an entire plant. Thankfully, that was before it was economical to put every controller on the Internet.

tamimio 2mo ago

The worst part -as I mentioned in another comment few days earlier- is most of the network is ancient, dialup, poor architecture designed by people who barely touched a computer, very old OSes, one utility had windows 3.1 a couple years ago! For them as long as it’s working, don’t touch it, because a downtime is far more costly and it will bring many eyes and attention to them than just sit and hope nothing will happen. Physical security is a major gap too, I have been to many locations where the field server panels are just behind the door, the RTUs are just closed with panel keys, even the whole facility are using some old HID cards that you can unlock in few seconds, this is even in R&D that beyond automation but also in robotic ones.

amluto 2mo ago

Another odd observation about the industry. For better or for worse, I inherited a little system controlled by a Fanuc PLC. It had a bug and would occasionally get stuck in a bad state. By some minor miracle, I managed to track down the person who had originally programmed it (now retired and moved out of state), and he emailed me a file and introduced me to his apprentice (who lives in a third state). I found a trial version of the design software, opened the file and found the bug (despite never having seen this style of programming before). Since I had no way to upload a new file (and since it wasn’t clear that uploading it without a matching outdated and not easily available copy of the programming environment would be wise), I convinced the apprentice to come visit while we was nearby for another client. He did not see the bug even when I pointed it out, but I convinced him to make the change and upload the modified file. Now the system works. (He had a Windows laptop with a giant folder of customer files, of course.) While he was on site, I asked him why there was nothing on the PLC or its enclosure to identify his old boss or give any contact information, and he seemed surprised and told me that no one ever does that. I asked how a new owner is supposed to get support, and he shrugged. If this thing ever fails, I’ll probably replace it with an Arduino or an ESP32 or something along those lines. Or I’ll just find something off the shelf to replace the entire system.

polishdude20 2mo ago
The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it.
- deleted 2mo ago
  [deleted]
- ssl-3 2mo ago
  You mean like, just hang a thumb drive on a lanyard inside of the cabinet -- with source code and such? That will probably just disappear. Maybe with good intention or maybe with bad, but it's probably gone either way. If it doesn't disappear, then it's hanging right there for any of the competitors to use. That's a problem for the original installer's ongoing employment. And if it includes the programming software, then that lets Joe (from over in shipping) have a go at rejiggering the packaging machine. That's a problem for whoever has to pay someone with a clue to show up and fix it.

procarch2019 2mo ago

Sounds like we have a similar development path. People fail to understand how complex some of these systems can be, especially since you have to have the project to understand what registers/variables are what. Some when Joe the contractor does a job and walks away, you better make sure they give you the project and you store it in a nice safe place. A good system integrator is worth their weight in gold. Sure they cost more, but getting a whole package turned over to you is worth a million more than 5 years into the lifecycle of a factory when someone wants to make mods/fix a bug/etc and has to reinvent the whole car, not just the wheel. This industry is always 10-20 years in the past. My company’s preferred vendor only just started supporting virtualization (this is for a DCS) in the past 10 years. I still have to tell my sales people to provide A/V and minimalistic backup and recovery on every project (they essentially cost nothing compared to the rest of any project).

closeparen 2mo ago
I enjoy watching Cursed Controls on YouTube. For him, the gold standard is a flash drive with the program on it physically hanging out in the control cabinet.

xtajv 2mo ago

Storytime: Once upon a time, I was a young grasshopper at a small consulting firm at which every employee was issued a Windows machine, our source control system was "\Customers" on C: drive, and our deployment strategy was "putty + copy-paste + hot-reload". One morning, I heard something terrible while my boss's boss's machine was booting. Something akin to grinding. The thing was angry. I gave boss^2 a heads-up that his hard drive might be failing, and that he might want to run a S.M.A.R.T. check on the poor thing. I also asked where the backup hard drives were, because I was brand-new and assumed that I just hadn't been issued one yet. I checked the supply closet, found no hard drives, popped over to the office admin person, and recommended a deal I'd seen on some WD black drives before I realized that the place had gone quiet. Everyone looked at me like I was from Mars. Exactly 7 days later, boss^2's hard drive failed. We lost a week of work and had to zero out a 5 days x 3 employees worth of billable hours. We also ended up delivering late. The client was pissed. Based on the nasty looks that I got afterwards, it appeared that the standard assumption was that I had tampered with the drive to prove a point. (Um, nope). I have since learned to ask prospective employers about their backup strategy.

MajorTakeaway 2mo ago
Nasty looks seems to be the sign of them realizing how you proved them to be incompetent. Incompetence will cost them their job. If I had fuck you money I'd just call them incompetent fucks and laugh at them.
- deleted 2mo ago
  [deleted]

bitwize 2mo ago

> This industry is in _desperate_ need of strong technical help, but unfortunately the "higher ups" tend to be the same people who are "comfortable" with the way thing are and refuse to move. I literally tried for a decade before giving up and moving into software engineering proper. As soon as AI can automate the development and deployment of physical infrastructure and the firmware that drives it, watch how willing to embrace the "cutting edge" those same higher-ups become.

jdw64 2mo ago

I agree. The problem is this: most companies using PLCs work in isolated networks with internet disconnected, based on confidentiality. There are many difficulties even with using git, because the structure forces you to implement features in a way that makes it hard to separate them. Branching, in particular, doesn't work well with PLC ladder logic—PLC uses proprietary binary formats that make diff tracking impossible, so you're forced into a unified development approach

PunchyHamster 2mo ago

I mean it worked fine for 50 years, surely it will work fine for another 50

tda 2mo ago

Yep, everything is lacking. No version control, ladder diagrams instead of written code, goto statements as primary control flow in code. A $500 PLC with less processing power than a 1985 086 and 1 whole MB of memory. Upgradable with a $200 license to 1.5MB, only programmable with a 20GB monstrosity of a piece of software in which download means copying a program.e to the PLC, and upload means downloading a backup to your laptop. That world needs some disruption really really bad

mikestew 2mo ago
Hey, hey, a little trigger warning over here? My last job before retirement was in that world. If it had been my first software job, it would have been my last.

KellyCriterion 2mo ago

May I ask: In which year was this? > I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID < The thing is: Big companies, even non critical are usually bonded to some frameworks/regulatory standards/etc. So, mentioning that they didnt know NAS or even RAID (which is very old, end of 80s), makes me wondering in which decade this happened?

structural 2mo ago
I've seen similar things in this decade. Lots of stuff still running where the last change in design was 20+ years ago, the current team knows how to keep the lights on and replace parts, and calls a vendor if something goes end of life and asks for a replacement part, then configures it the same way that the older one was. It's quite reliable, and does not require paying for a team of people who wants to try new things. Once this happens for a long while there's no one in the building left who's going to advocate for change, since the only thing all that fancy new technology brings is the next 5 years of nobody getting a bonus because operational metrics weren't being met anymore. You have to try really hard to convince these people that the thing that they have seen on a daily basis to work just fine for the past twenty years is no longer good enough, because all their evidence says that it is. And you should also listen to them much more than people who just want to use the latest thing typically do, because they do have the decades of experience seeing how things actually fail.

BrandoElFollito 2mo ago

I was on the cybersecurity side of that, "advising" huge, multinational companies that hold the world together. Oh boy. I was told for instance once that my concerns about an open, unencrypted Wi-Fi for SCADA systems were not a big deal because "we are in the desert anyway, who would come here". I saw terrible things and people who were made "cybersecurity expert" when they did not want to. They were sent to courses which were totally useless and the architecture was completely fucked up anyway. I often wondered over the years why we have so little significant incidents in the utilities sector.