5 ms·

It's fascinating how Raft has democratized distributed computing -- writing a consistent, distributed state machine now can be done in a few lines of code, assu

It's fascinating how Raft has democratized distributed computing -- writing a consistent, distributed state machine now can be done in a few lines of code, assuming you have a Raft implementation lying around, like Hashicorp's excellent Go library. What this project (and similar projects such as Rqlite) doesn't address is the hard problem -- sharding. Raft makes it quite trivial to write a master/slave system where every node is guaranteed to be identical, but it won't scale writes and won't distribute storage across your cluster. You can build sharding on top of Raft, but in an SQL setting this would require distributed transactions, something that's difficult enough that relatively few projects have tackled it so far, the notable ones being CockroachDB and TiDB. (Google Spanner, being proprietary, is not relevant in this context.)

eis 9y ago
> It's fascinating how Raft has democratized distributed computing First I thought you probably ment commoditized but then I realized how democratized actually applies perfectly well to consensus algorithms because by definition they apply decisions via voting by all participants :) But anyways, I agree very much with you that Raft has transformed the industry and we can't be thankful enough for it. But like you said, it's just one of the core building blocks of a distributed system that also shards data. Sharding is more use-case specific and so it can be valid for different databases to have different sharding logics.
- lobster_johnson 9y ago
  It's the second meaning of democratize, meaning "to make something accessible to everyone", that I intended, but I suppose Raft also has commoditized the technology, since there are now off-the-shelf libraries you can pick and choose from somewhat interchangeably. FWIW, the CockroachDB people developed a modified version they call MultiRaft [1] that deals with the scaling challenges they have around shards. [1] https://www.cockroachlabs.com/blog/scaling-raft/ https://www.cockroachlabs.com/blog/scaling-raft/
free-ekanayaka 9y ago
If you require SQL, sharding and concurrent writes, yes do use CockroachDB or other equivalent solution. If you don't have those requirements, dqlite or rqlite might get the job done with less moving parts and operational overhead. It's pretty much the same argument of using SQLite vs (say) PostgreSQL, but translated into distributed systems. See: https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html
jxub 9y ago
An excellent interactive explanation of Raft can be found here: http://thesecretlivesofdata.com/raft/ http://thesecretlivesofdata.com/raft/. It would be interesting as hell to build a small implementation in the language of choice (elixir?, pony?).
- usermac 9y ago
  That was good to see. Thank you.
zzzcpan 9y ago
> writing a consistent, distributed state machine now can be done in a few lines of code Well, if you run it on a very fast and reliable local network without much load, not multi dc deployments over public internet, than sure, it can sort of work. Not without problems though once a fault occurs.
- otoolep 9y ago
  I don't agree. The whole point of putting a solid Raft implementation at the center of your system is deal with the faults. I myself built a simple distributed state machine which deals with faults perfectly well. Of course I built on top of a good Raft implementation, which certainly is not a few lines of code. https://github.com/otoolep/hraftd https://github.com/otoolep/hraftd
  - zzzcpan 9y ago
    No, Raft doesn't deal with the faults, it deals with consensus and can help tolerate certain faults. You then deal with the faults by other means. And even on a fast and reliable local network where Raft can work well, once you have a significant amount of data replacing failing nodes, healing broken records, resyncing, rebalancing all without affecting operations is not trivial and unlikely to be done properly.
    - otoolep 9y ago
      > Raft doesn't deal with the faults I'm not sure what this means. The Raft paper explicitly states that Raft is a fault-tolerant system -- and by definition this means it deals with faults. To quote the paper: "Replicated state machines are used to solve a variety of fault tolerance problems in distributed systems." Raft is a type of replicated state machine. To say that "Raft doesn't deal with faults" is not correct. Perhaps you mean that there is other work to be done to take the fault tolerance offered by Raft and build a functioning application, and a system that will stay up in the real world -- and deal with an even wider range of faults. That I definitely agree with.
      - 9y ago
otoolep 9y ago
rqlite author here. I am considering adding distributed transaction support to rqlite (it already has a form of transaction support), if I can come up with a solid implementation, while keeping rqlite simple to deploy and operate (which is a primary goal of rqlite): https://groups.google.com/forum/#!topic/raft-dev/BaS5Z2NkDxA https://groups.google.com/forum/#!topic/raft-dev/BaS5Z2NkDxA https://github.com/rqlite/rqlite/issues/266 https://github.com/rqlite/rqlite/issues/266 dqlite is very interesting. If its patches to SQLite are accepted upstream, dqlite could become the storage engine for rqlite. But right now rqlite is deliberately built on vanilla-SQLite, so it can offer the same correctness guarantees as SQLite.
otoolep 9y ago
> What this project (and similar projects such as rqlite) doesn't address is the hard problem -- sharding. Yes, this is technically correct. However to meet its goals rqlite (and dqlites) does not require sharding. The point of rqlite is to provide fault-tolerance and high-availability. Again, you are technically correct that rqlite doesn't support sharding, but sharding simply isn't required for rqlite to do what it wants to do. Sharding is a solution to a different type of problem.
shalabhc 9y ago
Check out http://www.actordb.com/ http://www.actordb.com/ for a massively sharded sqlite system.
- lobster_johnson 9y ago
  ActorDB is very interesting. It uses two-phase commit, similar to CockroachDB, TiDB and Spanner, but as I understand it, relies on locking to update the participating shards. I'm not sure how well ActorDB performs in a large cluster with lots of distributed transactions, but I've never tried it.
  - shalabhc 9y ago
    IIUC these are micro shards - e.g. one sqlite instance per user. Depending on the key, distributed transactions may be rare.
    - atombender 9y ago
      The size of the shards depends on the data. ActorDB wants you to design your application around explicit containment boundaries (the actors). For example, a social networking site might collect all of a single user profile's data in one actor. Or there might be one actor for each grouping of a user's stuff (one actor for all of user X's photos, one actor for all of their tweets, etc.). So the size of those actors might not be "micro". You could decide to shard it further, of course, e.g. shard a user's photos by date. Since ActorDB supports cross-actor queries (though not joins), you can still query multiple shards at the same time. But ActorDB does seem to promote inter-actor transactions, and a pattern I've seen encouraged is where you use shared actors (especially with the key-value store functionality) to keep some kind of central lookup table that your app then can use to find actors. For example, if you're modeling HN with ActorDB, you'd have a global list of stories, but each story/comment tree could be a separate actor, and each user would be a separate actor. Posting a story only needs to access the story actor, but to post a comment transactionally, you have to do a transaction that inserts the comment and updates the user's comment history together.
jknoepfler 9y ago
is raft a fancy name for distinguished leader multipaxos?
- vertex-four 9y ago
  Raft does the things that Paxos does, using a mechanism that is much simpler for humans to understand and therefore implement correctly.
  - jknoepfler 9y ago
    if it does the things paxos does, it's provably isomorphic to paxos. what I saw in the demo was a flavor of paxos that's been around since the early 90's, I'm pretty sure that's where the credit is due.
    - vertex-four 9y ago
      You can't think of any case, for anything, in which you can do something in two (or more!) different ways, but one of them is simpler to explain, understand, and implement?

eis 9y ago

> It's fascinating how Raft has democratized distributed computing First I thought you probably ment commoditized but then I realized how democratized actually applies perfectly well to consensus algorithms because by definition they apply decisions via voting by all participants :) But anyways, I agree very much with you that Raft has transformed the industry and we can't be thankful enough for it. But like you said, it's just one of the core building blocks of a distributed system that also shards data. Sharding is more use-case specific and so it can be valid for different databases to have different sharding logics.

lobster_johnson 9y ago
It's the second meaning of democratize, meaning "to make something accessible to everyone", that I intended, but I suppose Raft also has commoditized the technology, since there are now off-the-shelf libraries you can pick and choose from somewhat interchangeably. FWIW, the CockroachDB people developed a modified version they call MultiRaft [1] that deals with the scaling challenges they have around shards. [1] https://www.cockroachlabs.com/blog/scaling-raft/ https://www.cockroachlabs.com/blog/scaling-raft/

free-ekanayaka 9y ago

If you require SQL, sharding and concurrent writes, yes do use CockroachDB or other equivalent solution. If you don't have those requirements, dqlite or rqlite might get the job done with less moving parts and operational overhead. It's pretty much the same argument of using SQLite vs (say) PostgreSQL, but translated into distributed systems. See: https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html

jxub 9y ago

An excellent interactive explanation of Raft can be found here: http://thesecretlivesofdata.com/raft/ http://thesecretlivesofdata.com/raft/. It would be interesting as hell to build a small implementation in the language of choice (elixir?, pony?).

usermac 9y ago
That was good to see. Thank you.

zzzcpan 9y ago

> writing a consistent, distributed state machine now can be done in a few lines of code Well, if you run it on a very fast and reliable local network without much load, not multi dc deployments over public internet, than sure, it can sort of work. Not without problems though once a fault occurs.

otoolep 9y ago
I don't agree. The whole point of putting a solid Raft implementation at the center of your system is deal with the faults. I myself built a simple distributed state machine which deals with faults perfectly well. Of course I built on top of a good Raft implementation, which certainly is not a few lines of code. https://github.com/otoolep/hraftd https://github.com/otoolep/hraftd
- zzzcpan 9y ago
  No, Raft doesn't deal with the faults, it deals with consensus and can help tolerate certain faults. You then deal with the faults by other means. And even on a fast and reliable local network where Raft can work well, once you have a significant amount of data replacing failing nodes, healing broken records, resyncing, rebalancing all without affecting operations is not trivial and unlikely to be done properly.
  - otoolep 9y ago
    > Raft doesn't deal with the faults I'm not sure what this means. The Raft paper explicitly states that Raft is a fault-tolerant system -- and by definition this means it deals with faults. To quote the paper: "Replicated state machines are used to solve a variety of fault tolerance problems in distributed systems." Raft is a type of replicated state machine. To say that "Raft doesn't deal with faults" is not correct. Perhaps you mean that there is other work to be done to take the fault tolerance offered by Raft and build a functioning application, and a system that will stay up in the real world -- and deal with an even wider range of faults. That I definitely agree with.
    - 9y ago

otoolep 9y ago

rqlite author here. I am considering adding distributed transaction support to rqlite (it already has a form of transaction support), if I can come up with a solid implementation, while keeping rqlite simple to deploy and operate (which is a primary goal of rqlite): https://groups.google.com/forum/#!topic/raft-dev/BaS5Z2NkDxA https://groups.google.com/forum/#!topic/raft-dev/BaS5Z2NkDxA https://github.com/rqlite/rqlite/issues/266 https://github.com/rqlite/rqlite/issues/266 dqlite is very interesting. If its patches to SQLite are accepted upstream, dqlite could become the storage engine for rqlite. But right now rqlite is deliberately built on vanilla-SQLite, so it can offer the same correctness guarantees as SQLite.

otoolep 9y ago

> What this project (and similar projects such as rqlite) doesn't address is the hard problem -- sharding. Yes, this is technically correct. However to meet its goals rqlite (and dqlites) does not require sharding. The point of rqlite is to provide fault-tolerance and high-availability. Again, you are technically correct that rqlite doesn't support sharding, but sharding simply isn't required for rqlite to do what it wants to do. Sharding is a solution to a different type of problem.

shalabhc 9y ago

Check out http://www.actordb.com/ http://www.actordb.com/ for a massively sharded sqlite system.

lobster_johnson 9y ago
ActorDB is very interesting. It uses two-phase commit, similar to CockroachDB, TiDB and Spanner, but as I understand it, relies on locking to update the participating shards. I'm not sure how well ActorDB performs in a large cluster with lots of distributed transactions, but I've never tried it.
- shalabhc 9y ago
  IIUC these are micro shards - e.g. one sqlite instance per user. Depending on the key, distributed transactions may be rare.
  - atombender 9y ago
    The size of the shards depends on the data. ActorDB wants you to design your application around explicit containment boundaries (the actors). For example, a social networking site might collect all of a single user profile's data in one actor. Or there might be one actor for each grouping of a user's stuff (one actor for all of user X's photos, one actor for all of their tweets, etc.). So the size of those actors might not be "micro". You could decide to shard it further, of course, e.g. shard a user's photos by date. Since ActorDB supports cross-actor queries (though not joins), you can still query multiple shards at the same time. But ActorDB does seem to promote inter-actor transactions, and a pattern I've seen encouraged is where you use shared actors (especially with the key-value store functionality) to keep some kind of central lookup table that your app then can use to find actors. For example, if you're modeling HN with ActorDB, you'd have a global list of stories, but each story/comment tree could be a separate actor, and each user would be a separate actor. Posting a story only needs to access the story actor, but to post a comment transactionally, you have to do a transaction that inserts the comment and updates the user's comment history together.

jknoepfler 9y ago

is raft a fancy name for distinguished leader multipaxos?

vertex-four 9y ago
Raft does the things that Paxos does, using a mechanism that is much simpler for humans to understand and therefore implement correctly.
- jknoepfler 9y ago
  if it does the things paxos does, it's provably isomorphic to paxos. what I saw in the demo was a flavor of paxos that's been around since the early 90's, I'm pretty sure that's where the credit is due.
  - vertex-four 9y ago
    You can't think of any case, for anything, in which you can do something in two (or more!) different ways, but one of them is simpler to explain, understand, and implement?