4 ms·
> This complex ownership model made it hard to move just a subset of user data to another region. Welcome to basically any large-scale enterprise. I have grow
by bob1029 5y ago
> This complex ownership model made it hard to move just a subset of user data to another region.
Welcome to basically any large-scale enterprise.
I have grown to learn that the active-passive strategy is the best option if the business can tolerate the necessary human delays. You get to side-step so much complicated bullshit with this path.
Trying to force active-active instant failover magic is sometimes not possible, profitable or even desirable. I can come up with a few scenarios where I would absolutely insist that a human go and check a few control points before letting another exact copy of the same system start up on its own, even if it would be possible in theory to have automatic failover work reliably 99.999% of the time.
- dboreham 5y agoMy fear with any sort of passive standby approach is that when the disaster comes, that standby won't work, or the mechanism used to fail over to it won't work. I prefer schemes where the "failover" is happening all the time hence I can be confident it works.
- orev 5y agoHaving a passive strategy doesn’t mean you don’t test it, and you can even perform actual failovers once on a while to validate everything. Active-active is also valid, but the point is that it comes with a huge amount of increased complexity. At some point you need to make a value calculation to decide if you want to focus on that, or on building the product.
- mike_d 5y agoA solid active/standby design should regularly flop. Every 2 weeks seems to be a sweet spot. This also balances wear across consumable hardware like disks. If your failover is happening "all the time" you basically just have a single system with failures.
- yencabulator 5y agoExcept for this: > However, because of that choice, replication between regions is asynchronous—meaning the remote replicas are always some number of transactions behind the primary region. Switching the active is a data loss event. Or, you can choose to drain the load and wait for the replication to catch up, but that's downtime and not a test of the real failure mechanism.
- deleted 5y ago[deleted]