3 ms·

I was just looking into this, and found this really interesting post about foundationdb and sql. https://www.voltactivedata.com/blog/2015/04/foundationdbs-less

I was just looking into this, and found this really interesting post about foundationdb and sql. https://www.voltactivedata.com/blog/2015/04/foundationdbs-lesson-fast-key-value-store-not-enough/ https://www.voltactivedata.com/blog/2015/04/foundationdbs-le... Also this, excellent comparison of distributed database consistency methods, mentions cockroach, but not foundation. http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-failing-to.html http://dbmsmusings.blogspot.com/2018/09/newsql-database-syst... This brings up the question, what method/protocol does foundation use for distributed consistency?

richieartoul 5y ago
"What method/protocol does foundation use for distributed consistency?" This is a really tough question to answer succinctly. It took me a long time to wrap my head around it, but the TLDR is: 1. It uses a quorum of Coordinator nodes to elect another node (via Paxos) whose job it is to then assigns roles to other nodes. 2. Some of those roles are replicated (Like TLogs) and some of those roles are singletons (like the thing that advances the clock and hands out versionstamps). 3. If any node in the "transaction subsystem" ever fails, the cluster goes through a recovery where it becomes unavailable for ~3s during which it reassigns the roles and keeps chugging. In terms of what it looks like for a transaction to actually commit, thats another long conversation. Its hard to draw analogies to existing systems because it really is kind of its own thing.
- ledgerdev 5y ago
  Awesome answer, thanks! Foundation seems like magic almost, from just choosing very smart trade offs and co-ordination.
  - richieartoul 5y ago
    Yeah that’s basically it. It really is a wonderful piece of free software that was (and still is) way ahead of its time in many ways. They basically made a ton of good architectural decisions really early on and then just nailed down the details super tight over many years. Their technique of running cluster-wide recoveries to handle failures is a great example. Most distributed systems engineers find the idea of designing a database that becomes unavailable (even for a few seconds) when a single node is lost completely appalling, but it turns out this small trade off leads to massive simplifications elsewhere and the resulting system is net/net much more reliable than anything else I’ve ever seen.

richieartoul 5y ago

"What method/protocol does foundation use for distributed consistency?" This is a really tough question to answer succinctly. It took me a long time to wrap my head around it, but the TLDR is: 1. It uses a quorum of Coordinator nodes to elect another node (via Paxos) whose job it is to then assigns roles to other nodes. 2. Some of those roles are replicated (Like TLogs) and some of those roles are singletons (like the thing that advances the clock and hands out versionstamps). 3. If any node in the "transaction subsystem" ever fails, the cluster goes through a recovery where it becomes unavailable for ~3s during which it reassigns the roles and keeps chugging. In terms of what it looks like for a transaction to actually commit, thats another long conversation. Its hard to draw analogies to existing systems because it really is kind of its own thing.

ledgerdev 5y ago
Awesome answer, thanks! Foundation seems like magic almost, from just choosing very smart trade offs and co-ordination.
- richieartoul 5y ago
  Yeah that’s basically it. It really is a wonderful piece of free software that was (and still is) way ahead of its time in many ways. They basically made a ton of good architectural decisions really early on and then just nailed down the details super tight over many years. Their technique of running cluster-wide recoveries to handle failures is a great example. Most distributed systems engineers find the idea of designing a database that becomes unavailable (even for a few seconds) when a single node is lost completely appalling, but it turns out this small trade off leads to massive simplifications elsewhere and the resulting system is net/net much more reliable than anything else I’ve ever seen.