4 ms·
There is almost never a single cause. If a single cause can trigger a disaster, then there is another cause by definition - poor system design. E.g. in the art
by janoc 6y ago
There is almost never a single cause. If a single cause can trigger a disaster, then there is another cause by definition - poor system design.
E.g. in the article's case it is clear that there is some sort of procedural deficiency there that allows the configuration variables to be set wrong and thus cause a connection to the wrong database.
Another one is that the function that has directly caused the data loss DOES NOT CHECK for this.
Yet another WTF is that if that code is meant to ever run on a development system, why is it in a production codebase in the first place?
And the worst bit? They throw arms up in the air, unable to identify the reason why this has happened. So they are leaving the possibility open to another similar mistake happening in the future, even though they have removed the offending code.
Oh and the fact that they don't have backups except for those of the hosting provider (which really shouldn't be relied on except as the last hail Mary solution!) is telling.
That's not a robust system design, especially if they are hosting customers' data.
- JackFr 6y agoThis should be a teachable moment with respect to their culture. Throwing up their hands without an understanding of what happened is unacceptable — if something that is believed impossible happens, it is important to know where your mental model failed. Otherwise you may make things worse by ‘remediating’ the wrong thing. And while this sounds overly simplistic the simplest way this could have been avoided is enforcing production hygiene. No developers on production boxes. Ever.
- deleted 6y ago[deleted]