3 ms·

Computers don't work, internet doesn't work, software doesn't work. If something functions flawlessly it's because there existed a group of people who figured o

Computers don't work, internet doesn't work, software doesn't work. If something functions flawlessly it's because there existed a group of people who figured out a rock solid setup at one point in time, and another group of people maintaining it actively right now. Many people seeing something that "just works" and it's "free as in beer" demand a similar setup for their own business assuming that creating it requires almost zero effort and cost.

csours 7y ago
A small example: A company develops a software product for a client. The company tests the product, and tests pass! No crashes! It just works. It works for years with no problems. Then you get a flurry of tickets in a pretty short window: the software is crashing! What happened? It passed tests, it worked for years! What changed in the environment? Are disks failing or filling up? No. Is the network failing? No. Is another program hogging resources? No. Is it a database maintenance window? No. Well the patch window changed, but smoke tests pass after patching, so it's not patches... Hmm, the patch window went from 2 weeks to 4 weeks. Ok, when did the software actually crash? About 25 days after patching. Is that consistent across sites? ... Yes it is. What happens 25 days after patching??? [Insert scramble to figure out what happens 25 days after patching] Is there anything special about 25 days? Well 2^31/86,400,000 is 24.85 ... Aka Signed Integer/Number of milliseconds in a day So all along there's been 24.85 day fuse in the code, but before the extension of the patch window we just happened to defuse the bomb every time. The supplier never had any reason to run a test for more than a week, so they never saw the crash either. ---- So, yes, software doesn't work. It happens to work right now, on this computer, with this hard drive, with these network settings, with these dependencies, with this database .... etc etc etc.
- sethammons 7y ago
  Fantastic story. Reminded me, we've seen similar issues back in the day when we were not monitoring memory usage as well as we should have been. Because we deployed daily or weekly, the services restarted and life was good. After everything was stable, a slow memory leak appeared that took weeks to grow to a point where the service would crash.
  - csours 7y ago
    We used a paging/alerting product. It was intended to be fired by command line, I guess, so each run was totally cleaned up because the process died. They added library (DLL) functionality so we could run it from a script. So the script ran and ran and ran and sent alerts, and randomly crashed from time to time. Why is it crashing? The stand-alone worked fine! Well, they weren't cleaning up their handles, so eventually we get to the process handle count limit and die!
brianyu8 7y ago
'The Dream Machine' by M. Mitchell Waldrop chronicles the trials and tribulations of the people who built the first computers and computer networks. Great read if you want to learn more about how computing and the internet were started!
- CBalin 7y ago
  This is the second time I've seen this book mentioned in just a few hours. Will definitely check it out.

csours 7y ago

A small example: A company develops a software product for a client. The company tests the product, and tests pass! No crashes! It just works. It works for years with no problems. Then you get a flurry of tickets in a pretty short window: the software is crashing! What happened? It passed tests, it worked for years! What changed in the environment? Are disks failing or filling up? No. Is the network failing? No. Is another program hogging resources? No. Is it a database maintenance window? No. Well the patch window changed, but smoke tests pass after patching, so it's not patches... Hmm, the patch window went from 2 weeks to 4 weeks. Ok, when did the software actually crash? About 25 days after patching. Is that consistent across sites? ... Yes it is. What happens 25 days after patching??? [Insert scramble to figure out what happens 25 days after patching] Is there anything special about 25 days? Well 2^31/86,400,000 is 24.85 ... Aka Signed Integer/Number of milliseconds in a day So all along there's been 24.85 day fuse in the code, but before the extension of the patch window we just happened to defuse the bomb every time. The supplier never had any reason to run a test for more than a week, so they never saw the crash either. ---- So, yes, software doesn't work. It happens to work right now, on this computer, with this hard drive, with these network settings, with these dependencies, with this database .... etc etc etc.

sethammons 7y ago
Fantastic story. Reminded me, we've seen similar issues back in the day when we were not monitoring memory usage as well as we should have been. Because we deployed daily or weekly, the services restarted and life was good. After everything was stable, a slow memory leak appeared that took weeks to grow to a point where the service would crash.
- csours 7y ago
  We used a paging/alerting product. It was intended to be fired by command line, I guess, so each run was totally cleaned up because the process died. They added library (DLL) functionality so we could run it from a script. So the script ran and ran and ran and sent alerts, and randomly crashed from time to time. Why is it crashing? The stand-alone worked fine! Well, they weren't cleaning up their handles, so eventually we get to the process handle count limit and die!

brianyu8 7y ago

'The Dream Machine' by M. Mitchell Waldrop chronicles the trials and tribulations of the people who built the first computers and computer networks. Great read if you want to learn more about how computing and the internet were started!

CBalin 7y ago
This is the second time I've seen this book mentioned in just a few hours. Will definitely check it out.