3 ms·

> Reproducibility matters > Immutable infrastructure removes a whole class of bugs. Story time. Back around 2010 I was tasked to figure out why a server appli

> Reproducibility matters > Immutable infrastructure removes a whole class of bugs. Story time. Back around 2010 I was tasked to figure out why a server application written in Java was running out of memory. It happened often enough that the application had to be restarted daily. I was not the one who maintained the application, and I wasn't the one maintaining the servers it ran on either. I was just someone good at solving these kinds of problems. The odd thing was that the problem only existed on one server in production, even though the other servers should be running the exact same thing. I ran load tests on a staging server, also running the same code, and was unable to reproduce the problem. With VisualVM and some heap dumps handy, I narrowed down the problem to database-related objects, specifically objects from the Oracle JDBC driver were taking the most memory. I tried checking what was different between the servers that would cause one to have an issue and not the others. Was one getting more load than the others? Were the processors different? Memory? Disk speed? To make a long story short, I spent entirely too long debugging the problem and eventually found out that one of the servers had a different version of the Oracle JDBC driver than the others. And it seemed that particular version of the driver had a memory leak bug. It took me so long because I assumed that nobody would have been dumb enough have different library files on different computers. I could kill whoever put that file on one server and not the others! It must have been done manually. I never did find out who did it or why. Immutable infrastructure would have prevented this problem, but this was back in 2010 where we ran our own servers and immutable infrastructure wasn't a possibility. If we had sufficiently reproducible deployments then we could have just redeployed the application and the problem would have gone away. Alas, the Oracle library wasn't part of the application code, but instead was part of library code that was installed on the computer and never changed (or so I thought). It's a godsend to be able to deploy containers that not only contain your application code, but also your server software and their dependent libraries. Reproducibility is so much better that way.

mjr00 4y ago
Yep, similar story as well from me, as I'm sure anyone who was doing software in the pre-cloud, pre-Docker era has... Worked at a consulting firm doing enhancements to a legacy hotel reservation system, and we had just finished our latest feature, which was integration with another, even more legacy reservation system (Visual FoxPro, yum). This was old-school consulting style, so we developed internally with our test systems and no access to the real systems, then shipped over an installer .exe and let the client, who ran ops, handle the installation and deployment into IIS. We built it, they deployed with no issue, test reservation worked, easy. A month later we got a report that about 20% of online reservations using this system were failing. Everything was working on our test systems, and it was working 80% of the time on their systems, so we assumed that it was a weird race condition with our booking service, which ran mostly asynchronously. We added a ton of additional logging to that service to try to figure out what was going on, sent them an updated version to deploy, and waited a bit. A week or so later I went over to the client's site to look at their production systems with their ops person. They had 5 servers servicing requests. We looked at servers 1-4, everything seemed fine. Open up server 5... wait, where's the log file? Process was running, IIS was responding, but nothing was coming out. Check the permissions on the folder, and lo and behold: the fifth server had a typo in the permission ACL entry for a folder needed for the legacy integration, which by fortunate coincidence was also where we had set the log files to write. Fixed the typo, the problem instantly went away completely. And being a consultant/client relationship, of course there was a lengthy legal dispute afterwards about whose fault this bug was, as that determined if it was covered under the original statement of work or we could charge more for it... one of the experiences that made me realize I needed to be working for a product company, not a consultancy.
ysleepy 4y ago
I've seen similar stuff, for me -XX:+HeapDumpOnOutOfMemoryError is a great thing, especially if set up to keep at max one dump to avoid out of disk errors. I feel managing dpendency jars separately was weird even in 2010, I always do a shaded jar or at least an maven assembly folder as a deployable to make sure the server runs the same combination as on my machine.

mjr00 4y ago

Yep, similar story as well from me, as I'm sure anyone who was doing software in the pre-cloud, pre-Docker era has... Worked at a consulting firm doing enhancements to a legacy hotel reservation system, and we had just finished our latest feature, which was integration with another, even more legacy reservation system (Visual FoxPro, yum). This was old-school consulting style, so we developed internally with our test systems and no access to the real systems, then shipped over an installer .exe and let the client, who ran ops, handle the installation and deployment into IIS. We built it, they deployed with no issue, test reservation worked, easy. A month later we got a report that about 20% of online reservations using this system were failing. Everything was working on our test systems, and it was working 80% of the time on their systems, so we assumed that it was a weird race condition with our booking service, which ran mostly asynchronously. We added a ton of additional logging to that service to try to figure out what was going on, sent them an updated version to deploy, and waited a bit. A week or so later I went over to the client's site to look at their production systems with their ops person. They had 5 servers servicing requests. We looked at servers 1-4, everything seemed fine. Open up server 5... wait, where's the log file? Process was running, IIS was responding, but nothing was coming out. Check the permissions on the folder, and lo and behold: the fifth server had a typo in the permission ACL entry for a folder needed for the legacy integration, which by fortunate coincidence was also where we had set the log files to write. Fixed the typo, the problem instantly went away completely. And being a consultant/client relationship, of course there was a lengthy legal dispute afterwards about whose fault this bug was, as that determined if it was covered under the original statement of work or we could charge more for it... one of the experiences that made me realize I needed to be working for a product company, not a consultancy.

ysleepy 4y ago

I've seen similar stuff, for me -XX:+HeapDumpOnOutOfMemoryError is a great thing, especially if set up to keep at max one dump to avoid out of disk errors. I feel managing dpendency jars separately was weird even in 2010, I always do a shaded jar or at least an maven assembly folder as a deployable to make sure the server runs the same combination as on my machine.