3 ms·
I worked on a service that had this exact issue; actually it was worse because it was a signed integer so rolled over in half the time. Even more insidious, we
by ryanschneider 7y ago
I worked on a service that had this exact issue; actually it was worse because it was a signed integer so rolled over in half the time.
Even more insidious, we never hit the issue until months into production because we had a pretty consistent two week release process. So the issue only came to light when development slowed down as the project became more stable.
Luckily deployments were staggered across datacenters so when the bug hit it didn’t happen to every server at once, but of course as always happens, when it did, the majority of the servers started hanging over the weekend while I was on call.
- yjftsjthsd-h 7y agoI worked at a company with a similar issue. The server is used to get patched and rebooted every month. Then the company decided to lay off huge chunks of operations staff. The software held up remarkably well, but we eventually discovered that software that was never designed to stay up for more than a month will leak memory like crazy if you leave it up for a long enough.
- sn_master 7y agoIIS used (still does?) to randomly kill its own processes at random because chances are, if its been running "long enough" its leaking memory anyway.