Yep, similar story as well from me, as I'm sure anyone who was doing software in the pre-cloud, pre-Docker era has... Worked at a consulting firm doing enhance

Yep, similar story as well from me, as I'm sure anyone who was doing software in the pre-cloud, pre-Docker era has... Worked at a consulting firm doing enhancements to a legacy hotel reservation system, and we had just finished our latest feature, which was integration with another, even more legacy reservation system (Visual FoxPro, yum). This was old-school consulting style, so we developed internally with our test systems and no access to the real systems, then shipped over an installer .exe and let the client, who ran ops, handle the installation and deployment into IIS. We built it, they deployed with no issue, test reservation worked, easy. A month later we got a report that about 20% of online reservations using this system were failing. Everything was working on our test systems, and it was working 80% of the time on their systems, so we assumed that it was a weird race condition with our booking service, which ran mostly asynchronously. We added a ton of additional logging to that service to try to figure out what was going on, sent them an updated version to deploy, and waited a bit. A week or so later I went over to the client's site to look at their production systems with their ops person. They had 5 servers servicing requests. We looked at servers 1-4, everything seemed fine. Open up server 5... wait, where's the log file? Process was running, IIS was responding, but nothing was coming out. Check the permissions on the folder, and lo and behold: the fifth server had a typo in the permission ACL entry for a folder needed for the legacy integration, which by fortunate coincidence was also where we had set the log files to write. Fixed the typo, the problem instantly went away completely. And being a consultant/client relationship, of course there was a lengthy legal dispute afterwards about whose fault this bug was, as that determined if it was covered under the original statement of work or we could charge more for it... one of the experiences that made me realize I needed to be working for a product company, not a consultancy.