3 ms·
The problem happens over time when both properties begin to fail after changes are made to the system by people unfamiliar with the system, and make worse when
by coldcode 4y ago
The problem happens over time when both properties begin to fail after changes are made to the system by people unfamiliar with the system, and make worse when sudden failures are addressed by the same people, which creates a feedback loop of failures. Eventually the system becomes unusable. Generally companies don't lay off huge swatches of people randomly, so this kind of bit rot is fairly rare.
My last employer one day had virtually every system fail in the world at the same time due to a hidden dependency on some routing in a single data center; it took 8 hours to restart all the systems since that had not happened before. It was a fun slack to follow. There was a lot of institutional knowledge that was able to get everything working again. Imagine this kind of issue with people having no idea how things work...
- spookthesunset 4y ago> My last employer one day had virtually every system fail in the world at the same time due to a hidden dependency on some routing in a single data center; it took 8 hours to restart all the systems since that had not happened before. It was a fun slack to follow. I think every company eventually has at least once of these. It is usually a wake up call to the company that what used to work when they had N number of users no longer scales with 10x the traffic… (It also usually has a lot of “I told you so” moments too)