4 ms·

After reading the paper, I felt disappointed that: 1) Bizur is used for building a scalable object storage rather than a filesystem, which does not even support

After reading the paper, I felt disappointed that: 1) Bizur is used for building a scalable object storage rather than a filesystem, which does not even support atomic rename for single key (the hash of key will change, it will be moved to another bucket without any transactional guarantee) 2) Bizur itself looks like a sharded Raft group, which is already used in TiDB (one raft group per region). Please correct me if read it wrongly.

ideal0227 10y ago
This (like a sharded Raft group) is somewhat correct :P. If you can sacrifice consistency (causal instead of external), you can get EPaxos. If you can sacrifice more, you can get Single Paxos. Both of them can increase concurrency to some degree, and can improve availability over a leader based RSM system like Muti-paxos or Raft. I believe there are other ways to make the trade-off, even making the trade-off dynamically. Hopefully, I can talk about it more soon. I work on etcd/raft, which is a shared Raft implementation by TiDB(TiKV), Cockroachdb, etcd and many more.
- rystsov 10y ago
  > If you can sacrifice more, you can get Single Paxos What do you mean? Single Decree Paxos is linearizable so it's consistent. When the multi-key updates are necessary - they can be implemented with client-side transactions such as 2PC, RAMP or Percolator. Even with Raft we in the same situation. Once the data overgrows the size of one machine we need to shard them. As a result some keys can live on different shards and we need cross shards transactions to do multi-key updates :)
  - ideal0227 10y ago
    Well... Why do you need 2PC if nothing is sacrificed? :) If you choose things like Raft (or other RSM), you can get a consistent key space to do txn over keys. For a metadata store like etcd, one RSM group is OK. Performance/availability is better than something requires 2PC. For database systems, there are quite a few txns that can actually fall into one shard and can bypass 2PC. Of course, it is highly depend on the workload and use cases.
rystsov 10y ago
The paper is quite important - it's the first paper about the production usage of an algorithm similar to Single-Decree Paxos. 1) The atomicity usually isn't a problem, it can be added on top of the sharded storage with 2PC, RAMP or Percolator transactions, or with Sagas. 2) With Raft you have to implement log compaction and snapshotting, with Bizur and Single-Decree Paxos it isn't the case
ezrahoch 10y ago
One of the differences between a raft group and a bizur bucket is the granularity and the overhead of that granularity. Bizur buckets can be very small (10s of kb), and the overhead for them is minimal (10s of bytes). What would be the overhead of having a raft group for every 32KB? Would it be practical / a good use of the machine's resources?

ideal0227 10y ago

This (like a sharded Raft group) is somewhat correct :P. If you can sacrifice consistency (causal instead of external), you can get EPaxos. If you can sacrifice more, you can get Single Paxos. Both of them can increase concurrency to some degree, and can improve availability over a leader based RSM system like Muti-paxos or Raft. I believe there are other ways to make the trade-off, even making the trade-off dynamically. Hopefully, I can talk about it more soon. I work on etcd/raft, which is a shared Raft implementation by TiDB(TiKV), Cockroachdb, etcd and many more.

rystsov 10y ago
> If you can sacrifice more, you can get Single Paxos What do you mean? Single Decree Paxos is linearizable so it's consistent. When the multi-key updates are necessary - they can be implemented with client-side transactions such as 2PC, RAMP or Percolator. Even with Raft we in the same situation. Once the data overgrows the size of one machine we need to shard them. As a result some keys can live on different shards and we need cross shards transactions to do multi-key updates :)
- ideal0227 10y ago
  Well... Why do you need 2PC if nothing is sacrificed? :) If you choose things like Raft (or other RSM), you can get a consistent key space to do txn over keys. For a metadata store like etcd, one RSM group is OK. Performance/availability is better than something requires 2PC. For database systems, there are quite a few txns that can actually fall into one shard and can bypass 2PC. Of course, it is highly depend on the workload and use cases.

rystsov 10y ago

The paper is quite important - it's the first paper about the production usage of an algorithm similar to Single-Decree Paxos. 1) The atomicity usually isn't a problem, it can be added on top of the sharded storage with 2PC, RAMP or Percolator transactions, or with Sagas. 2) With Raft you have to implement log compaction and snapshotting, with Bizur and Single-Decree Paxos it isn't the case

ezrahoch 10y ago

One of the differences between a raft group and a bizur bucket is the granularity and the overhead of that granularity. Bizur buckets can be very small (10s of kb), and the overhead for them is minimal (10s of bytes). What would be the overhead of having a raft group for every 32KB? Would it be practical / a good use of the machine's resources?