7 ms·

The concept of the "global log" is mentioned many times in the article, but not much detail on this critical piece: > ... the consensus protocol is only being

by fernly 8y ago

The concept of the "global log" is mentioned many times in the article, but not much detail on this critical piece: > ... the consensus protocol is only being used for inserting transactions into a global log. For every other part of the protocol, replicas can proceed completely independently from each other. Clearly this central resource can't be distributed or replicated. Right? So it, and the hardware it runs on, and the communications links to it, are a single point of failure. True? If the global log goes down, or can't be reached at all, or perhaps only from some region of the regionized application, what happens?

jchrisa 8y ago
The global log is replicated using a Raft-like consensus mechanism, so that all replicas have a copy of the log, and agree on its order. So even if one replica site is lost, the others have a full copy of the log, and can continue. You can learn more about Raft from this excellent visualization: http://thesecretlivesofdata.com/raft/ http://thesecretlivesofdata.com/raft/
jamesblonde 8y ago
You are correct that they serialize cross-partition transactions. So if your workload has a reasonable percentage of write/update/delete operations it is possible that you will bottleneck on the global coordinator. From this blog https://fauna.com/blog/distributed-acid-transaction-performance https://fauna.com/blog/distributed-acid-transaction-performa..., you can get 3300 transactions/second. Daniel Abadi claims you can get to 500,000 trans/sec - http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-failing-to.html http://dbmsmusings.blogspot.com/2018/09/newsql-database-syst.... In any case, there is an open-source in-memory database called NDB (MySQL Cluster) that can get to 100 million write transactions per second. The echo chamber of SV has ignored it, but it's real: http://mikaelronstrom.blogspot.com/2015/03/200m-reads-per-second-in-mysql-cluster.html http://mikaelronstrom.blogspot.com/2015/03/200m-reads-per-se... NDB has what i call lock-aware programming. When you read a row, you take either a read_lock or a write_lock, as it only supports READ_COMMITTED isolation level. But, it does not serialize cross-partition transactions. Transaction coordinators run independently, and it is up to the developer to ensure consistency by taking either read or write locks on rows.
- jchanimal 8y ago
  The log segments committed by FaunaDB contain batches of transactions, which means our throughput is constrained not by our consensus protocol, but rather by conflicts as transactions are resolved. The benchmarked 3300 transactions/second mentioned is for complex transactions with dozens of reads and writes. Additionally, read-only transactions are not run through the transaction pipeline, since they can be served consistently with low latency from replicas using FaunaDB's snapshot storage. More important for most applications than theoretical benchmarks is taking a sound approach and using a database you can depend on. Asking your development team to manage isolation is going to introduce more costs than benefits, except in the most specialized of circumstances. FaunaDB's model is always ACID, so you never have to guess what's in the database, even in the presence of region failures or other anomalies.
  - jamesblonde 8y ago
    We built HopsFS on NDB, not CalvinDB, because we needed performance for cross-partition transactions. Some workloads need it. In the filesystem workload, when FS path components are normalized and stored on different partitions, practically all transactions cross partitions. So if you serialize them, then writing a file in /home/jchan will block writing a file in /home/jim. This is what most distributed filesystems already do - have a global write lock on the filesystem metadata. I like having the freedom to build my the concurrency model on top of READ_COMMITTED, as i can squeeze out 16X performance improvemnts by doing so (ref: https://www.usenix.org/conference/fast17/technical-sessions/presentation/niazi https://www.usenix.org/conference/fast17/technical-sessions/... )
    - evanweaver 8y ago
      Sure, if you don't care about serializability, you don't need to pay consensus costs to enforce it, but you will be subject to read and write skew.
    - notacoward 8y ago
      > This is what most distributed filesystems already do - have a global write lock on the filesystem metadata Absolutely untrue. Stop lying about other systems.
evanweaver 8y ago
The "global log" is a logical component. It is a distributed, partitioned write-ahead log which is implemented in a pipelined version of Raft. It is both distributed and replicated, even across datacenters, depending on topological configuration. We will update the post to make this more clear.

by fernly 8y ago

jchrisa 8y ago

The global log is replicated using a Raft-like consensus mechanism, so that all replicas have a copy of the log, and agree on its order. So even if one replica site is lost, the others have a full copy of the log, and can continue. You can learn more about Raft from this excellent visualization: http://thesecretlivesofdata.com/raft/ http://thesecretlivesofdata.com/raft/

jamesblonde 8y ago

You are correct that they serialize cross-partition transactions. So if your workload has a reasonable percentage of write/update/delete operations it is possible that you will bottleneck on the global coordinator. From this blog https://fauna.com/blog/distributed-acid-transaction-performance https://fauna.com/blog/distributed-acid-transaction-performa..., you can get 3300 transactions/second. Daniel Abadi claims you can get to 500,000 trans/sec - http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-failing-to.html http://dbmsmusings.blogspot.com/2018/09/newsql-database-syst.... In any case, there is an open-source in-memory database called NDB (MySQL Cluster) that can get to 100 million write transactions per second. The echo chamber of SV has ignored it, but it's real: http://mikaelronstrom.blogspot.com/2015/03/200m-reads-per-second-in-mysql-cluster.html http://mikaelronstrom.blogspot.com/2015/03/200m-reads-per-se... NDB has what i call lock-aware programming. When you read a row, you take either a read_lock or a write_lock, as it only supports READ_COMMITTED isolation level. But, it does not serialize cross-partition transactions. Transaction coordinators run independently, and it is up to the developer to ensure consistency by taking either read or write locks on rows.

jchanimal 8y ago
The log segments committed by FaunaDB contain batches of transactions, which means our throughput is constrained not by our consensus protocol, but rather by conflicts as transactions are resolved. The benchmarked 3300 transactions/second mentioned is for complex transactions with dozens of reads and writes. Additionally, read-only transactions are not run through the transaction pipeline, since they can be served consistently with low latency from replicas using FaunaDB's snapshot storage. More important for most applications than theoretical benchmarks is taking a sound approach and using a database you can depend on. Asking your development team to manage isolation is going to introduce more costs than benefits, except in the most specialized of circumstances. FaunaDB's model is always ACID, so you never have to guess what's in the database, even in the presence of region failures or other anomalies.
- jamesblonde 8y ago
  We built HopsFS on NDB, not CalvinDB, because we needed performance for cross-partition transactions. Some workloads need it. In the filesystem workload, when FS path components are normalized and stored on different partitions, practically all transactions cross partitions. So if you serialize them, then writing a file in /home/jchan will block writing a file in /home/jim. This is what most distributed filesystems already do - have a global write lock on the filesystem metadata. I like having the freedom to build my the concurrency model on top of READ_COMMITTED, as i can squeeze out 16X performance improvemnts by doing so (ref: https://www.usenix.org/conference/fast17/technical-sessions/presentation/niazi https://www.usenix.org/conference/fast17/technical-sessions/... )
  - evanweaver 8y ago
    Sure, if you don't care about serializability, you don't need to pay consensus costs to enforce it, but you will be subject to read and write skew.
  - notacoward 8y ago
    > This is what most distributed filesystems already do - have a global write lock on the filesystem metadata Absolutely untrue. Stop lying about other systems.

evanweaver 8y ago

The "global log" is a logical component. It is a distributed, partitioned write-ahead log which is implemented in a pipelined version of Raft. It is both distributed and replicated, even across datacenters, depending on topological configuration. We will update the post to make this more clear.