3 ms·
To make it work with any kind of performance, all message passing needs to be asynchronous, and you need an event loop to process messages one at a time. Things
by ccleve 8y ago
To make it work with any kind of performance, all message passing needs to be asynchronous, and you need an event loop to process messages one at a time. Things are much simpler if your code is effectively single-threaded when processing an inbound message. That took me a while to figure out. Futures and promises are very helpful.
It took a while to get the building blocks together: the log file, which can be traversed, read, and rolled back/truncated (use a kv store for this, trust me), and the transport mechanism to deal with sending complex objects and handling errors and timeouts.
Tracking terms, states (leader, follower, candidate), rejecting out of date terms, voting, elections -- it's all very complex. You'll have to build some really, really good multi-threaded tests that simulate out-of-order and dropped messages, then really pound away at the system. And good logging to find that odd corner case that keeps coming up and throwing mysterious messages.
I haven't even tried membership changes yet.
- bch 8y ago> use a kv store for this, trust me Order is important here - is this inherent in your key?
- ccleve 8y agoYes. I did two implementations, one with plain text files and one with a kv store. For plain text files, I had to maintain file offsets of the records so I could traverse backward and truncate records that had to be replaced, and then break the files into segments so I could retire segments that are no longer needed. It had all kinds of little bugs. Then I reimplemented using a kv store, and it took a tiny fraction of the time. And surprisingly had better performance. Just create a good interface at the start, use the kv store underneath, and then later if you don't want the dependency swap it out for the more complicated code.