3 ms·

A small example: A company develops a software product for a client. The company tests the product, and tests pass! No crashes! It just works. It works for yea

by csours 7y ago

A small example: A company develops a software product for a client. The company tests the product, and tests pass! No crashes! It just works. It works for years with no problems. Then you get a flurry of tickets in a pretty short window: the software is crashing! What happened? It passed tests, it worked for years! What changed in the environment? Are disks failing or filling up? No. Is the network failing? No. Is another program hogging resources? No. Is it a database maintenance window? No. Well the patch window changed, but smoke tests pass after patching, so it's not patches... Hmm, the patch window went from 2 weeks to 4 weeks. Ok, when did the software actually crash? About 25 days after patching. Is that consistent across sites? ... Yes it is. What happens 25 days after patching??? [Insert scramble to figure out what happens 25 days after patching] Is there anything special about 25 days? Well 2^31/86,400,000 is 24.85 ... Aka Signed Integer/Number of milliseconds in a day So all along there's been 24.85 day fuse in the code, but before the extension of the patch window we just happened to defuse the bomb every time. The supplier never had any reason to run a test for more than a week, so they never saw the crash either. ---- So, yes, software doesn't work. It happens to work right now, on this computer, with this hard drive, with these network settings, with these dependencies, with this database .... etc etc etc.

sethammons 7y ago
Fantastic story. Reminded me, we've seen similar issues back in the day when we were not monitoring memory usage as well as we should have been. Because we deployed daily or weekly, the services restarted and life was good. After everything was stable, a slow memory leak appeared that took weeks to grow to a point where the service would crash.
- csours 7y ago
  We used a paging/alerting product. It was intended to be fired by command line, I guess, so each run was totally cleaned up because the process died. They added library (DLL) functionality so we could run it from a script. So the script ran and ran and ran and sent alerts, and randomly crashed from time to time. Why is it crashing? The stand-alone worked fine! Well, they weren't cleaning up their handles, so eventually we get to the process handle count limit and die!

by csours 7y ago

sethammons 7y ago

Fantastic story. Reminded me, we've seen similar issues back in the day when we were not monitoring memory usage as well as we should have been. Because we deployed daily or weekly, the services restarted and life was good. After everything was stable, a slow memory leak appeared that took weeks to grow to a point where the service would crash.

csours 7y ago
We used a paging/alerting product. It was intended to be fired by command line, I guess, so each run was totally cleaned up because the process died. They added library (DLL) functionality so we could run it from a script. So the script ran and ran and ran and sent alerts, and randomly crashed from time to time. Why is it crashing? The stand-alone worked fine! Well, they weren't cleaning up their handles, so eventually we get to the process handle count limit and die!