7 ms·

ACID transactions in a globally distributed database

123 points by jchanimal 9y ago

alexvoda 9y ago
Interesting but ultimately irrelevant since it appears to not be FLOSS. We have already seen what happened to FoundationDB.
- deleted 9y ago
  [deleted]
- setra 9y ago
  They are primarily targeting the cloud database market. There are many examples of non free software surviving just fine in this market. Both amazon and google run their own proprietary cloud databases like this.
  - jankotek 9y ago
    And it is very easy to host your own instance of Amazon or Google :-)
smnscu 9y ago
I started using CockroachDB [1] recently and I'm really impressed with it overall. The Postgres compatibility is genius in my opinion, and despite some hurdles it's been a joy to work with (I'm using xo [2] to generate Go code from custom templates). I've yet to deploy it in production, but the best part is that if the performance sucks I can simply deploy postgres instead. As someone who used to be a big RethinkDB supporter, this is my new favorite pet oss project. edit: oh, and it's written in Go, my main language, which is definitely a plus if you're a gopher 1: https://www.cockroachlabs.com/tags/acid/ https://www.cockroachlabs.com/tags/acid/ 2: https://github.com/xo/xo https://github.com/xo/xo
- therealmarv 9y ago
  Postgres compatibility? Really? Reference? I want to believe but are not you comparing apples to oranges? Last time I checked the JSON datatype was not supported on Cockroach.
  - detaro 9y ago
    CockroachDB explicitly aims to offer a Postgres-compatible interface. Is their homepage reference enough for that? https://www.cockroachlabs.com/product/cockroachdb/ https://www.cockroachlabs.com/product/cockroachdb/ or https://www.cockroachlabs.com/docs/stable/build-an-app-with-cockroachdb.html https://www.cockroachlabs.com/docs/stable/build-an-app-with-...
    - therealmarv 9y ago
      no, not even a JSON datatype is now supported. I cannot use a database without JSON for my use case https://github.com/cockroachdb/cockroach/issues/2969 https://github.com/cockroachdb/cockroach/issues/2969 An compatible interface is not compatibility and JSON as an example is not the biggest magic stuff inside PostgreSQL.
      - ddorian43 9y ago
        Give the dudes time. Postgres has a lot of stuff.
        orangechairs 9y ago
        JSONB support is on our 2.0 roadmap and is actively under development. Roadmap: https://github.com/cockroachdb/cockroach/wiki/Roadmap https://github.com/cockroachdb/cockroach/wiki/Roadmap Confirmed Development (I see you already reference this issue above): https://github.com/cockroachdb/cockroach/issues/2969 https://github.com/cockroachdb/cockroach/issues/2969
      - smnscu 9y ago
        JSONB support is coming, fuck yeah! https://github.com/cockroachdb/docs/issues/2143 https://github.com/cockroachdb/docs/issues/2143
mabbo 9y ago
> industry experience has taught us that availability—in the strict sense that it is defined in the CAP theorem—is overrated: In practice, the uptime of systems that favor availability have not proven greater than what can be achieved with consistent systems. The answer to any distributed consistent database is always the same: we gave up on one letter in CAP. In this case, it was 'A', availability. And hey, cool, that's certainly an option. But I'd prefer if they had simply made that the subtitle.
- jacobparker 9y ago
  What they're trying to explain is that so-called AP systems do not have 100% availability either. The CAP theorem applies to very specific scenarios that are not necessarily common - it is a much more narrow theorem then "CP vs AP" implies. For example, if your network partitions are usually small (i.e. there exists a majority outside the partition) and you can avoid routing requests to the minority partition (e.g. because you peer with ISPs and can route somewhere else, or internally avoid routing to a rack that is don etc.) then you won't observe a loss of A. For more information on this perspective you can read _Spanner, TrueTime and the CAP theorem_ by the fella that coined the term CAP: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45855.pdf https://static.googleusercontent.com/media/research.google.c... Yammer concurred with this perspective, stating: "At Yammer we have experience with AP systems, and we’ve seen loss of availability for both Cassandra and Riak for various reasons. Our AP systems have not been more reliable than our CP systems, yet they have been more difficult to work with and reason about in the presence of inconsistencies. Other companies have also seen outages with AP systems in production. So in practice, AP systems are just as susceptible as CP systems to outages due to issues such as human error and buggy code, both on the client side and the server side." https://yokota.blog/2017/02/17/dont-settle-for-eventual-consistency/ https://yokota.blog/2017/02/17/dont-settle-for-eventual-cons...
  - pkolaczk 9y ago
    "In practice, the uptime of systems that favor availability have not proven greater than what can be achieved with consistent systems." This claim is much stronger than just saying AP systems sometimes fail. Therefore observing some AP systems failures is not enough to prove it. For the statement to hold true, AP systems would have to fail just as often as CP systems. However, such claim is totally unfounded. The article doesn't link to any research showing AP systems have the same (or worse) availability than CP systems. On the other hand, most CP systems, even centralized ones, are not truly consistent as required by ACID either. Most RDBMS systems run at Read Committed level for performance reasons, which is ACD, not ACID. It is also ignoring the fact that most systems are neither AP or CP, and that this distinction is not very useful to describe database systems.
d__k 9y ago
In a globally distributed database, the main problem is time control because different locations have different clocks. For example, assume that one table of the database is located on Earth while another table is stored on Mars (4-13 minutes delay). Now we want to commit a transaction with these two tables from the moon while the result will be queried from Venus. I could not understand from this post if Fauna is able to correctly work under such conditions and whether it was designed for such use cases.
- sleepydog 9y ago
  "Globally distributed" should imply that it's confined to one planet.
  - pkolaczk 9y ago
    Even then, the roundtrip time between continents can be in range of several hundreds milliseconds, which is far too much for many real time transactional systems.
  - mangatmodi 9y ago
    @sleepydog, that was an example. Even in one planet, RTT in internet can be quite severe for realtime applications.
  - vidarh 9y ago
    More generally, the point is that if you have two locations where the fastest you can send a unit of data is t, then the shortest possible time interval you can pass a transaction from one location to another is t. That puts a upper (EDIT: corrected from lower) bound on throughput of many types of operations, such as e.g. if you need to guarantee strict ordering of transactions (EDIT: in the worst case, where transactions are being issued from multiple nodes; best case you can optimize in a variety of ways). So while we won't have to deal with interplanetary latencies, it still matters greatly for throughput and consistency guarantees.
    - mattb314 9y ago

123 points by jchanimal 9y ago

Interesting but ultimately irrelevant since it appears to not be FLOSS. We have already seen what happened to FoundationDB.

deleted 9y ago
[deleted]
setra 9y ago
They are primarily targeting the cloud database market. There are many examples of non free software surviving just fine in this market. Both amazon and google run their own proprietary cloud databases like this.
- jankotek 9y ago
  And it is very easy to host your own instance of Amazon or Google :-)

smnscu 9y ago

I started using CockroachDB [1] recently and I'm really impressed with it overall. The Postgres compatibility is genius in my opinion, and despite some hurdles it's been a joy to work with (I'm using xo [2] to generate Go code from custom templates). I've yet to deploy it in production, but the best part is that if the performance sucks I can simply deploy postgres instead. As someone who used to be a big RethinkDB supporter, this is my new favorite pet oss project. edit: oh, and it's written in Go, my main language, which is definitely a plus if you're a gopher 1: https://www.cockroachlabs.com/tags/acid/ https://www.cockroachlabs.com/tags/acid/ 2: https://github.com/xo/xo https://github.com/xo/xo

therealmarv 9y ago
Postgres compatibility? Really? Reference? I want to believe but are not you comparing apples to oranges? Last time I checked the JSON datatype was not supported on Cockroach.
- detaro 9y ago
  CockroachDB explicitly aims to offer a Postgres-compatible interface. Is their homepage reference enough for that? https://www.cockroachlabs.com/product/cockroachdb/ https://www.cockroachlabs.com/product/cockroachdb/ or https://www.cockroachlabs.com/docs/stable/build-an-app-with-cockroachdb.html https://www.cockroachlabs.com/docs/stable/build-an-app-with-...
  - therealmarv 9y ago
    no, not even a JSON datatype is now supported. I cannot use a database without JSON for my use case https://github.com/cockroachdb/cockroach/issues/2969 https://github.com/cockroachdb/cockroach/issues/2969 An compatible interface is not compatibility and JSON as an example is not the biggest magic stuff inside PostgreSQL.
    - ddorian43 9y ago
      Give the dudes time. Postgres has a lot of stuff.
      - orangechairs 9y ago
        JSONB support is on our 2.0 roadmap and is actively under development. Roadmap: https://github.com/cockroachdb/cockroach/wiki/Roadmap https://github.com/cockroachdb/cockroach/wiki/Roadmap Confirmed Development (I see you already reference this issue above): https://github.com/cockroachdb/cockroach/issues/2969 https://github.com/cockroachdb/cockroach/issues/2969
    - smnscu 9y ago
      JSONB support is coming, fuck yeah! https://github.com/cockroachdb/docs/issues/2143 https://github.com/cockroachdb/docs/issues/2143

mabbo 9y ago

> industry experience has taught us that availability—in the strict sense that it is defined in the CAP theorem—is overrated: In practice, the uptime of systems that favor availability have not proven greater than what can be achieved with consistent systems. The answer to any distributed consistent database is always the same: we gave up on one letter in CAP. In this case, it was 'A', availability. And hey, cool, that's certainly an option. But I'd prefer if they had simply made that the subtitle.

jacobparker 9y ago
What they're trying to explain is that so-called AP systems do not have 100% availability either. The CAP theorem applies to very specific scenarios that are not necessarily common - it is a much more narrow theorem then "CP vs AP" implies. For example, if your network partitions are usually small (i.e. there exists a majority outside the partition) and you can avoid routing requests to the minority partition (e.g. because you peer with ISPs and can route somewhere else, or internally avoid routing to a rack that is don etc.) then you won't observe a loss of A. For more information on this perspective you can read _Spanner, TrueTime and the CAP theorem_ by the fella that coined the term CAP: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45855.pdf https://static.googleusercontent.com/media/research.google.c... Yammer concurred with this perspective, stating: "At Yammer we have experience with AP systems, and we’ve seen loss of availability for both Cassandra and Riak for various reasons. Our AP systems have not been more reliable than our CP systems, yet they have been more difficult to work with and reason about in the presence of inconsistencies. Other companies have also seen outages with AP systems in production. So in practice, AP systems are just as susceptible as CP systems to outages due to issues such as human error and buggy code, both on the client side and the server side." https://yokota.blog/2017/02/17/dont-settle-for-eventual-consistency/ https://yokota.blog/2017/02/17/dont-settle-for-eventual-cons...
- pkolaczk 9y ago
  "In practice, the uptime of systems that favor availability have not proven greater than what can be achieved with consistent systems." This claim is much stronger than just saying AP systems sometimes fail. Therefore observing some AP systems failures is not enough to prove it. For the statement to hold true, AP systems would have to fail just as often as CP systems. However, such claim is totally unfounded. The article doesn't link to any research showing AP systems have the same (or worse) availability than CP systems. On the other hand, most CP systems, even centralized ones, are not truly consistent as required by ACID either. Most RDBMS systems run at Read Committed level for performance reasons, which is ACD, not ACID. It is also ignoring the fact that most systems are neither AP or CP, and that this distinction is not very useful to describe database systems.

d__k 9y ago

In a globally distributed database, the main problem is time control because different locations have different clocks. For example, assume that one table of the database is located on Earth while another table is stored on Mars (4-13 minutes delay). Now we want to commit a transaction with these two tables from the moon while the result will be queried from Venus. I could not understand from this post if Fauna is able to correctly work under such conditions and whether it was designed for such use cases.

sleepydog 9y ago
"Globally distributed" should imply that it's confined to one planet.
- pkolaczk 9y ago
  Even then, the roundtrip time between continents can be in range of several hundreds milliseconds, which is far too much for many real time transactional systems.
- mangatmodi 9y ago
  @sleepydog, that was an example. Even in one planet, RTT in internet can be quite severe for realtime applications.
- vidarh 9y ago
  More generally, the point is that if you have two locations where the fastest you can send a unit of data is t, then the shortest possible time interval you can pass a transaction from one location to another is t. That puts a upper (EDIT: corrected from lower) bound on throughput of many types of operations, such as e.g. if you need to guarantee strict ordering of transactions (EDIT: in the worst case, where transactions are being issued from multiple nodes; best case you can optimize in a variety of ways). So while we won't have to deal with interplanetary latencies, it still matters greatly for throughput and consistency guarantees.
  - mattb314 9y ago