1. As long as active dataset everything fits in RAM, performance will be great. E.g. you can have terabytes of data but as long as the actively accessed dataset

1. As long as active dataset everything fits in RAM, performance will be great. E.g. you can have terabytes of data but as long as the actively accessed dataset is < ~80GB the system will perform well. Once things get out of RAM, everything will work well if you have an SSD. On a rotational disk, performance will degrade very very quickly. This is a bottleneck on all modern databases, but clustering makes this problem go away because you can effectively increase the amount of RAM at linear cost by just adding more nodes (e.g. two nodes at 100GB of RAM each cost about four times less than one node with 200GB of RAM). 2. We do do block-level replication. On each node of the btree we store replication timestamps. When a node asks for new data, we can cull away parts of the tree the node has almost instantly. So replication is very very efficient for most OLTP workloads. We don't have statement-level replication yet, so if you do a range update on a large table, we'll have to replicate data block by block. It'll take a while to add statement-based replication - we'd have to do a pretty significant refactoring to make it happen. 3. Either. Replicas are great for failover -- if the master dies, you just failover and a replica picks up where the master left off. If you're ok with out-of-date reads, you can also hit replicas directly (e.g. for reports, etc.) and spread out the read load across the cluster. 4. This is a really complex question - we didn't document this because doing it properly would take a lot of time. I'll ask jdoliner to chime in -- he designed the architecture and wrote most of the code, perhaps he can describe it succinctly while we write deeper docs on this :) 5. We use protocol buffers between the client drivers and the server. We picked that because there were libraries for the initial three languages we picked (Ruby, JS, Python), they were really easy to use, and very efficient. We could also have a single spec for the client/server API. Internally we use our own serialization scheme which allows us to dump arbitrary C++ objects on the network. It doesn't support other languages (which we didn't need), but is much more versatile for writing complex cross-machine code.