3 ms·

Could a partition just be the machine hosting one of your DBs going down? When it comes back up, it still has its data, but it's missed out on what happened in

Could a partition just be the machine hosting one of your DBs going down? When it comes back up, it still has its data, but it's missed out on what happened in the meantime.

robertlagrant 2y ago
Not really - if a database stops you fundamentally don't have availability. CAP is specific to network partitions because it wants to grapple with what happens when clients can talk to nodes in your service, but the nodes can't talk to each other.
hinkley 2y ago
Your question was answered by someone else but you've triggered a different form of dread. We lost a preprod database that held up dev and QA for half a day. To bring the server room up to code they moved the outlets, and when they went to move the power cords, someone thought they could get away with relying on the UPS power during the jiggery. Turned out the UPS that was reporting all green only actually had about 8 seconds of standby power in it, which was a little longer than the IT person needed to untangle the cables. So in the end we were just fucked on a different day than we would have been if we had waited for the next hiccup in the city power. I mention this because if you've split your cluster between three racks and someone manages to power down an entire rack, you're dangerously close to not having quorum and definitely close to not having sufficient throughput for both consumer traffic and rebuilding/resyncing/resilvering. It's a slow motion rock slide that is a cluster that was not sized correctly for user traffic plus recovery traffic as the entire thing limps toward a full on outage when just one more machine needs to be rebooted, because, for instance, your team decided that restart a machine every n/48 hours, to deal with a slow leak is just fine since you're using a consensus protocol that will hide the reboots. Maybe rock slide is the wrong word. It's a train wreck. Once it starts you can't stop it, you can only watch and fret.

Could a partition just be the machine hosting one of your DBs going down? When it comes back up, it still has its data, but it's missed out on what happened in the meantime.

robertlagrant 2y ago

Not really - if a database stops you fundamentally don't have availability. CAP is specific to network partitions because it wants to grapple with what happens when clients can talk to nodes in your service, but the nodes can't talk to each other.

hinkley 2y ago

Your question was answered by someone else but you've triggered a different form of dread. We lost a preprod database that held up dev and QA for half a day. To bring the server room up to code they moved the outlets, and when they went to move the power cords, someone thought they could get away with relying on the UPS power during the jiggery. Turned out the UPS that was reporting all green only actually had about 8 seconds of standby power in it, which was a little longer than the IT person needed to untangle the cables. So in the end we were just fucked on a different day than we would have been if we had waited for the next hiccup in the city power. I mention this because if you've split your cluster between three racks and someone manages to power down an entire rack, you're dangerously close to not having quorum and definitely close to not having sufficient throughput for both consumer traffic and rebuilding/resyncing/resilvering. It's a slow motion rock slide that is a cluster that was not sized correctly for user traffic plus recovery traffic as the entire thing limps toward a full on outage when just one more machine needs to be rebooted, because, for instance, your team decided that restart a machine every n/48 hours, to deal with a slow leak is just fine since you're using a consensus protocol that will hide the reboots. Maybe rock slide is the wrong word. It's a train wreck. Once it starts you can't stop it, you can only watch and fret.