4 ms·
Clustering and automatic fail-over and recovery is not always a great idea. It sounds like you want it implemented so you "don't have to worry about it", but fa
by ploxiln 10y ago
Clustering and automatic fail-over and recovery is not always a great idea. It sounds like you want it implemented so you "don't have to worry about it", but fail-over and recovery systems are a lot more complicated and troublesome than an alert, some read-only degraded operation, and a manual promotion.
Aphyr's "call me maybe" series shows just how hard it is to get automatic failover and recovery working just right. If you're not ready to become a full-time expert on the topic, and pay for more servers for the same load, then simple replication and manual promotion can result in less downtime and less dataloss. I've seen multiple small startups have cassandra clusters fail, because they did not maintain them properly. I've seen them mysteriously lose data in elasticsearch clusters, for the same reason.
That said, GitLab has recently shown that even simple replication can be screwed up. I'd say, the lesson is, the most effective thing is to keep it simple (not "easy"), and avoid disaster caused by mundane mistakes.
- ansible 10y agoYeah, if you can wait another year or two, something like CockroachDB [1] might fit the bill. That is a database that is distributed from the start. So you (theoretically) won't have to worry as much about individual datacenter failures, and scaling. [1] http://www.cockroachlabs.com http://www.cockroachlabs.com
- StreamBright 10y agoI usually wait 5 years with a database/datastore after it goes GA before consider it for anything. Looks promising though.
- ansible 10y agoYes, I've liked the design from the very start, and they've got a nice group of people working on it. It's supposed to go into a 1.0 release this year. I'll probably try it out on a service where I'm writing simultaneously to two different databases, one local and one distributed. And then log any issues where I see an inconsistency. I won't be relying on it for a while for anything critical.
- brightball 10y agoWhat about Couchbase?
- StreamBright 10y agoWe had serious operational issues with Couchbase and some funny developers where always ready to answer our tickets with "works on my laptop". One we had 1.3 billion documents in the cluster and it became unstable and we had to delete all of the data to bring the service back online. Luckily we used S3 as the source of truth based on the operational characteristics of CB. I would not recommend it for anybody who is not ready for massive dataloss and continuous operational issues. Some of these might be been fixed in the meanwhile though.
- throwaway91111 10y agoCouchbase appears to require failover as it has authoritative servers (three, it looks like?). CockroachDB should scale linearly with cluster size, meaning failover isn't a thing--it just struggles until you give it more CPU/disk/mem/what have you.
- phamilton 10y agoMy opinion is evolving on this matter. I've seen a more than a couple of developer introduced bugs in our application that cause data loss (improper error handling, swallowing errors instead of retrying, etc.) We use RDS with automatic failover and redislabs cluster with automatic failover. I'm 99% positive we have lost data before, and it makes me very uncomfortable to think about. But since we lose data from buggy code, I really can't really justify a huge amount of engineering effort to avoid the chance of loss in DB failover.