3 ms·
2000x multiplier between connectivity loss and service degradation.
by why_only_15 8y ago
2000x multiplier between connectivity loss and service degradation.
- tatersolid 8y agoThis is rather common; too many HA technologies really aren’t designed for wide-area failover, or anything besides the total and complete failure of a node. I once saw a ten-second “grey failure” in the network cause an entire SAN cluster to go read-only for a day while it checksummed all replicas. If you don’t Chaos-monkey test these things in production, they really won’t work when needed. And you can’t just “unplug a box” in your testing, you need to test slowdowns and intermittent errors under production loads. Few enterprises are willing to risk or invest in that sort of testing. So we all hope for the best and point fingers upstream at vendors when the HA doesn’t work as advertised.
- jeremyjh 8y agoI've suffered more outages at the hands of HA database technologies (that were administered in-house) than such have prevented. There are very few organizations I trust to run them properly.
- tatersolid 8y agoI’ve always stuck with log shipping to a warm standby wherever possible for the same reason. Well-managed single instance databases can do 3+ nines easily; very few applications actually require more availability than that. Especially at 5x the cost in opex for a theoretical-but-not-really-wink-wink extra 9.
- jrochkind1 8y agoIn github's case, of course, they can't do a single instance db for performance reasons, even before you get to HA. If I understand it right.
- jeremyjh 8y agoThroughput is orthogonal to reliability. You can improve both by creating read replicas, but that doesn't imply automated fail-over.
- manquer 8y agoDepends on how your architecture, github as a whole likely has volume of writes greater than a single server instance, however either components could be split in separate dbs ( seems to be their approach) or shard / partition the data