2 ms·
The main difference being the time it takes to recover/reverse the decision. Second point is: You don't need to reverse the decision you took, instead you may
by pvtmert 8mo ago
The main difference being the time it takes to recover/reverse the decision.
Second point is: You don't need to reverse the decision you took, instead you may find a way to fix the impact but not the root-cause.
It's like when one fucks up the MySQL replication and the data consistency is corrupted. One can manually (and slowly) fix the inconsistency with downtime. Or, spin up a whole new cluster from an existing well-known node/state. Some entities may be missing, but you could gradually add them back later.
Not a reversible, but recoverable decision.
Amazon goes by with one-way vs two-way door decisions internally. Sometimes adding much bureaucracy to the equation. Just-do-it/Bias-for-action aspect usually don't go as far as the recovery period prolongs.
- actionfromafar 8mo agoYes - when reading what large entities do and don't, one should always reality check when applying for your situation: do I have a billion dollars earmarked for recovery?
- pjc50 8mo agoI'm reminded of the Lloyds/TSB IT failure, where they specifically didn't have a rollback plan and ended up getting fined £50m by the regulator: https://www.bbc.co.uk/news/business-64036529 https://www.bbc.co.uk/news/business-64036529