At my previous company we experimented with a 12 node cluster (128GB per node) used as a datastore. At one point we had over 1 billion keys. This was back in th

At my previous company we experimented with a 12 node cluster (128GB per node) used as a datastore. At one point we had over 1 billion keys. This was back in the 1.x version where they persisted data into sqlite files so each file would have 100+ million rows in it. Persisting data took forever. Rebalancing took forever. When it worked, it was very fast. When you lost a node, things went bad. The java client would lose it's mind trying to cope with an outage. We ended up writing a connection pool were we could just recreate our connections when we detected a node went out. That said, it was the best distributed NoSQL solution we tried and might have improved a lot since I last used it.