4 ms·
Key takeaway for me: "crash-restart durable storage is a prerequisite for most consensus algorithms, I have a hunch we’ll see stronger guarantees as Elasticsea
by jodah 11y ago
Key takeaway for me:
"crash-restart durable storage is a prerequisite for most consensus algorithms, I have a hunch we’ll see stronger guarantees as Elasticsearch moves towards consensus on writes."
It seems like ElasticSearch is inching towards a proper consensus algorithm, at least optionally, which makes me wonder yet again, why not just implement RAFT? While I won't speculate I will point out that the answer for other systems appears to have been related to ego (ie, distributed systems are easy).
- jpgvm 11y agoThe funny thing is there is a community plugin that fixes almost all of this by using Zookeeper for discovery and master election. The not so funny thing is Elastic employed the primary developer of said plugin and basically shut it down.
- kimchy 11y agoactually, it doesn't fixes the mentioned bug. I am only posting this here to make sure people won't go and try and install it thinking it does... There is a difference between cluster level master election and replication semantics in Elasticsearch (and other similar systems). Even if you use Zookeeper for cluster level master election, one still needs to handle point to point replication (which doesn't go through zk).
- jpgvm 11y agoI should have been more specific. The ZK plugin fixes the issues related to network split-brains and other nasty partition conditions. The newly discovered translog issue is also a problem and is not solved by ZK but is also a lot less scary.