9 ms·

Serializable, Lockless, Distributed: Isolation in CockroachDB

130 points by orangechairs 10y ago

tyingq 10y ago
I love the transparency this project operates with. Looking forward to when joins are implemented so I can try using it with something non-trivial.
- chillaxtian 10y ago
  > Looking forward to when joins are implemented so I can try using it with something non-trivial. these types of databases typically don't have joins. you'll be waiting a while.
  - tyingq 10y ago
    It's not a nosql database if that's what you're thinking. https://github.com/cockroachdb/cockroach/issues/2970 https://github.com/cockroachdb/cockroach/issues/2970
    - andrewvc 10y ago
      Joins in distributed databases all must make some sort of unsavory tradeoff be it speed or space or limiting what you can join. So, while any distributed database can do a join, it may not be fast or flexible enough to be worth it.
      - wichsen 10y ago
        You are right. And without data and performance numbers there is no way to extrapolate beyond "potential" scenarios and outcomes. Please forgive me, but what are you really trying to get at? Otherwise it just seems like naysaying and rabble rousing to me..
      - teraflop 10y ago
        One could say exactly the same thing about joins in non-distributed databases. "You have to make some kind of tradeoff" is a platitude.
      - biokoda 10y ago
        Joins in ActorDB work great. Of course that is because we use an entirely different way of making an SQL database distributed and our joins aren't actually distributed even though the database is. Best way to solve a problem is to avoid it.
        tyingq 10y ago
        I looked at ActorDB, with the thought of using it as a sort of "Sql Enabled" version of etcd. Meaning, storing config data, mostly reads, with no big performance requirements. But, the use case is to ensure that the config data is available on all nodes....high availability. So, for example, sharding isn't wanted or needed. It was difficult, however, to get my arms around the whole "actor model", and understand how to use ActorDB in this relatively simple deployment. Long story short, is there some reference, or dead simple example of a "single actor" deployment with a small number of tables?
        biokoda 10y ago
        It's no different from any other deployment really. You have a single actor type in your init.sql then when you send in queries you always specify the same actor. It's easier to explain if you tell me what you don't understand.
        tyingq 10y ago
        It's basically trying to figure out what the purpose of an "actor" is, and what it means in terms of schema design. I can't tell if I'm supposed to use multiple actors because it adds some kind of resilience, or performance, or something else? On the surface, it seems analogous to "CREATE DATABASE" / "USE DATABASE", but then the examples show applications using multiple actors...which wouldn't be typical (one app / multiple databases). So, it seems clear the idea is multiple actors within a single app, but what drives the choice of what the actors are? If the documentation started with some deeper explanation of actors, it might be easier to follow...as is, it jumps into actor syntax, creation, etc, without the reader really knowing what one is first. I do get that this might be unique to me...I'm just not grocking the concept.
  - mjibson 10y ago
    Joins are coming, although they will take some time.
  - teraflop 10y ago
    CockroachDB doesn't currently support joins, but it's been designed so that it can potentially support them, unlike most NoSQL databases. In particular, it supports cross-machine transactions, which are a critical building block for both correctness and performance. For instance, suppose you want to join on a column that's not a primary key. No big deal in a typical RDBMS; just do an index lookup on the join column. But if your data is sharded across machines, that doesn't work, because you can't update the index without running the risk that readers will observe it in an inconsistent state. Unless you can do transactions, that is. So I'm optimistic that Cockroach will get join support before too long. The necessary primitives are available, making it "merely" a matter of figuring out the engineering details.
    - sriram_malhar 10y ago
      Yes, distributed joins are well understood. But I'm not at all convinced that the relational model is a good fit for a scale-out, widely distributed, fault-tolerant database. The engineering decisions for each of these aspects add enough latency to the process as it is.
  - petermattis 10y ago
    (employee/founder here) Re joins: we have top men working on it right now. Joins are definitely an interesting problem for a distributed database.
    - chris_wot 10y ago
      Hi Peter, sounds pretty awesome! Quick question though - on your front page you say that CockroachDB does SQL - but if it can't do a join, then how can you say it uses SQL? Or is distributed SQL a different thing entirely? It does sounds like a very limited SQL subset though... I'm sure I must be missing something as I'm not familiar with your product. Also, what levels of isolation do you actually offer? Serialized snapshot isolation appears to be MVCC, but I see you also have just snapshot isolation - what is the difference? Edit: oh brother, the proof you link to, I just realized I bought that book some time ago and never got around to reading it... Transactional Information Systems by Weikum & Vossen, right? time for me to hit the books I guess. Still trying to get my head around that second graph you have drawn, can't work out how you have gotten it :(
      - knz42 10y ago
        SQL is not a single language but rather a family of languages. Each RDBMS that advertises support for SQL ends up implementing a different flavor of SQL, most of the times they are not even compatible :) For now CockroachDB's supports a subset of the SQL implemented by other databases, with some extensions of its own. This subset will grow over time. The two levels of isolation offered are snapshot (SI) and serializable (SSI). Snapshot means that concurrent transactions are atomic with regards to each other and "see" the same initial state from the DB. Serializable adds into this that they can't introduce write skews, ie if there are concurrent transactions they will "see" each others effects in some order.
      - petermattis 10y ago
        Hi Chris, we describe CockroachDB as a SQL database because that is what we're aspiring to. The missing functionality (i.e. joins) is on our near-term roadmap.
  - jandrewrogers 10y ago
    It is possible to massively distribute joins but it requires a much more sophisticated database design than you see in most distributed database implementations because the whole system has to be designed for that use case e.g. support for ad hoc inter-node orchestration and data flows. At that point you basically have a parallel database instead of just a distributed one. And if you add an additional requirement to correctly execute ad hoc consistent joins under very high write workloads and continuous failures, then it is definitely non-trivial engineering. But it is possible, I've designed implementations like this before, it just requires a very high degree of effort and skill that is rarely applied to the design of typical NoSQL databases.
  - manigandham 10y ago
    MemSQL does distributed joins really well, using a combination of in-memory rowstore + disk columnstore and the ability to have certain tables replicated to every node for faster joining.
danieltoshea 10y ago
How does this work if the clocks drift between the nodes? Does this allow incorrect behavior because one transaction looks like it happened before another?
- knz42 10y ago
  This was discussed earlier on the same blog: https://www.cockroachlabs.com/blog/living-without-atomic-clocks/ https://www.cockroachlabs.com/blog/living-without-atomic-clo... Short answer: the DB will check for drifts and correct small drifts automatically.
- mrtracy 10y ago
  (blog author here) Interestingly, clock drift does not affect the serializability of the transaction history; this system guarantees that the history is serializable, regardless of clock drift. However, "serializable" only means that the history is equivalent to some serial ordering of transactions - it makes no guarantee that the equivalent serial ordering is consistent with the real-time ordering of the involved transactions. A history with that property (agrees with real-time) is termed "linearizable", and requires additional rules to guarantee in an environment with clock drift. As mentioned by knz42, there was another Cockroach Labs blog post (written by Spencer Kimball) that addressed this in some detail; that blog post contrasted our strategy for dealing with drift with that of Google's Spanner. A quick overview of CockroachDB's properties re linearizability: it guarantees that access to any individual key is linearizable, and by composition any two transactions which share a key (that one of the transactions modifies) will be linearizable with respect to each other. However, if two transactions do not have any overlap in modified keys, Cockroach does not (by default) guarantee the resulting commit history is linearizable. CockroachDB's underlying KV layer does have a "linearizable" flag on transactions that can guarantee this, but it requires that transactions be slowed down considerably; Spencer's blog post addresses some other strategies that CockroachDB is considering to address the issue.
  - lobster_johnson 10y ago
    Can Cockroach do the equivalent of a "select ... for update" (e.g., PostgreSQL), where you lock one thing while applying changes elsewhere? Concrete example: We have app that has a "documents" table and a "translog" table. The translog is like a series of diff-patches, representing changes to the documents. When we write to the translog, we first lock the document with a "select ... for update", so that no intervening translog entries can be written concurrently against the same document, then we patch the document, and then we write the translog entry and commit. We do this with Postgres, and we can do the same thing with Redis' MULTI since Redis is completely single-threaded. I can't think of any other NoSQL data store that allows a similar "lock A, update A, insert B, unlock A"; for example, Cassandra's "lightweight transactions" are only transactional in the context of a single row. (By "lock" I'd also accept optimistic locking, where you can retry on failure.)
barking 10y ago
Sorry if this is a little bit off topic This is a startup with employees that is making a non-commercial open source database system. How will they make a financial return on such a product?
- sgarman 10y ago
  It's common in the industry to build an open source product and then do paid support, hosting, etc with it. Joyent/Node Sonatype Nexus
  - barking 10y ago
    OK thanks, I sort of thought that it might be something like that but wondered if I was missing some other angle.
LoSboccacc 10y ago
By coincidence I was looking around for a datastore. I almost settled on couchbase when I saw this; easy cluster deployment is my main draw after looking at how bad setting up master-master system is in other solutions (hbase, mongo, couchdb all have eterogeneus nodes and weird failure modes) One quick question since I saw the devs around and I can't find a final answer on it on google: are there any strong roadblock or performance drawbacks against storing principally medium size binary data like images, say, < 100mb blobs? Yeah I could use something more appropriate but I'd be back to figuring out hadoop installation or fighting s3 eventual consistency.
- ansible 10y ago
  I don't know that you would be happy trying to store blobs of that size in any distributed database, current or future. I'd suggest having an immutable distributed blob store, and just store the index in the distributed database. Of course, now you are half way towards a distributed filesystem...
  - LoSboccacc 10y ago
    but then you'd need to store file path in the datastore, with all the failure modes that follows. I know what the current best practices are, but for bootstrapped one man startups you always end up with dozen moving parts to be tamed and instead of building stuff you need to solve the same problems over and over again I'm implementing a datastore on top of glusterfs, which while amateurish is a wonderful learning experience, but since it's nowhere close to being workable I am also looking around for what's happening in this space. I've this longer term goal about having a serverless framework where the datastore also provide oauth for users, which was inspired by https://github.com/serverless/serverless https://github.com/serverless/serverless - would be a boon for client side webapp app style startups.
    - manigandham 10y ago
      I have a strong sense that you're vastly over-engineering something. The solved problem is called an object store. Just use S3 or Azure Storage or Google Cloud Storage - they all have options for geo-replication and it'll be just as fast or faster than any database replication you set up, while being far more 9's of availability and reliability. Storing a tiny bit of text for the filename is trivial and you can use any durable database for that.
grogers 10y ago
Naively it looks like your conflict resolution is prone to livelock, where concurrent transactions for a key keep aborting each other without getting any work done. The same can happen if you run a single node RDBMS in serializable isolation mode and read a row before writing to it. In that case you can add locking reads on rows you intend to later write to, which avoids the livelock. Have you done any studies to quantify how bad that effect is in cockroachdb? Assuming the effect exists and I didn't just miss something silly, are there any workarounds since cockroachdb is only OCC? (Exponential backoff between retries and keeping transactions short don't count, I assume those are a given)
- knz42 10y ago
  When transactions conflict, the priority of the losing transaction is internally ratcheted up. At some point it's just higher than everything after it and it succeeeds.
- mrtracy 10y ago
  (author) This is a good insight; Live Lock was one of the things that kept me up at night, although my fellow contributors have allayed my concerns considerably. In a write-heavy workload where the transactions follow a read-modify-write pattern, it does seem possible that your application could be facing live lock concerns. So, let's look at where those concerns come from in CockroachDB: 1. For write-write conflicts, live-lock concerns comes from the fact that a later transaction can abort your in-progress transaction if it has a priority. On top of this, if your transaction ends up aborting an earlier transaction, the other transaction may retry with a potentially higher priority, and this can result in a "priority war" of sorts. 2. "Read-write" conflicts have a worse problem: if a transaction with a later timestamp reads a key before your transaction writes to it, then your transaction always aborts. In read-modify-write workloads where the read is first, this seems to be the biggest theoretical source of live-lock. Now, as for dealing with those issues. + The "Write-write" issue is the smaller concern here. The ratcheting-up of priorities is probabilistic; currently, after enough retries your transaction will have a high enough priority that it will almost certainly make progress. However, we are not totally content with this: we have an issue for our 1.0 release (https://github.com/cockroachdb/cockroach/issues/5727 https://github.com/cockroachdb/cockroach/issues/5727) to do some serious investigation of this, with some alternatives already suggested. For example, one such alternative is using "lowest original timestamp wins" to settle conflicts instead of a random priority. + The read-write concern seems more problematic; our read timestamp cache retains minimal data due to memory concerns, and thus aborts a bit conservatively (i.e. aborts transactions that might have been able to continue). This can be dealt with by modifying your application pattern - in particular, I would suggest using our "SNAPSHOT" isolation level for many workloads. I didn't cover this in the blog post, but in short the SNAPSHOT mode allows RW conflicts to occur without aborting transactions. SNAPSHOT transactions are subject to the "write skew" anomaly, but this anomaly does not occur if the involved transactions write to a common key. This is fortunately the case for many OLTP-type workloads, meaning SNAPSHOT can be used without anomalies; this also moves conflict-detection responsibility entirely to WW conflicts, which abort less conservatively than RW and should ameliorate any live-lock problems.
ruggeri 10y ago
I just wanted to thank the team for this writeup. I've been excited to read it since the last post several months back. This was definitely very useful, and very clearly written. So many thanks!

130 points by orangechairs 10y ago

I love the transparency this project operates with. Looking forward to when joins are implemented so I can try using it with something non-trivial.

chillaxtian 10y ago
> Looking forward to when joins are implemented so I can try using it with something non-trivial. these types of databases typically don't have joins. you'll be waiting a while.
- tyingq 10y ago
  It's not a nosql database if that's what you're thinking. https://github.com/cockroachdb/cockroach/issues/2970 https://github.com/cockroachdb/cockroach/issues/2970
  - andrewvc 10y ago
    Joins in distributed databases all must make some sort of unsavory tradeoff be it speed or space or limiting what you can join. So, while any distributed database can do a join, it may not be fast or flexible enough to be worth it.
    - wichsen 10y ago
      You are right. And without data and performance numbers there is no way to extrapolate beyond "potential" scenarios and outcomes. Please forgive me, but what are you really trying to get at? Otherwise it just seems like naysaying and rabble rousing to me..
    - teraflop 10y ago
      One could say exactly the same thing about joins in non-distributed databases. "You have to make some kind of tradeoff" is a platitude.
    - biokoda 10y ago
      Joins in ActorDB work great. Of course that is because we use an entirely different way of making an SQL database distributed and our joins aren't actually distributed even though the database is. Best way to solve a problem is to avoid it.
      - tyingq 10y ago
        I looked at ActorDB, with the thought of using it as a sort of "Sql Enabled" version of etcd. Meaning, storing config data, mostly reads, with no big performance requirements. But, the use case is to ensure that the config data is available on all nodes....high availability. So, for example, sharding isn't wanted or needed. It was difficult, however, to get my arms around the whole "actor model", and understand how to use ActorDB in this relatively simple deployment. Long story short, is there some reference, or dead simple example of a "single actor" deployment with a small number of tables?
        biokoda 10y ago
        It's no different from any other deployment really. You have a single actor type in your init.sql then when you send in queries you always specify the same actor. It's easier to explain if you tell me what you don't understand.
        tyingq 10y ago
        It's basically trying to figure out what the purpose of an "actor" is, and what it means in terms of schema design. I can't tell if I'm supposed to use multiple actors because it adds some kind of resilience, or performance, or something else? On the surface, it seems analogous to "CREATE DATABASE" / "USE DATABASE", but then the examples show applications using multiple actors...which wouldn't be typical (one app / multiple databases). So, it seems clear the idea is multiple actors within a single app, but what drives the choice of what the actors are? If the documentation started with some deeper explanation of actors, it might be easier to follow...as is, it jumps into actor syntax, creation, etc, without the reader really knowing what one is first. I do get that this might be unique to me...I'm just not grocking the concept.
- mjibson 10y ago
  Joins are coming, although they will take some time.
- teraflop 10y ago
  CockroachDB doesn't currently support joins, but it's been designed so that it can potentially support them, unlike most NoSQL databases. In particular, it supports cross-machine transactions, which are a critical building block for both correctness and performance. For instance, suppose you want to join on a column that's not a primary key. No big deal in a typical RDBMS; just do an index lookup on the join column. But if your data is sharded across machines, that doesn't work, because you can't update the index without running the risk that readers will observe it in an inconsistent state. Unless you can do transactions, that is. So I'm optimistic that Cockroach will get join support before too long. The necessary primitives are available, making it "merely" a matter of figuring out the engineering details.
  - sriram_malhar 10y ago
    Yes, distributed joins are well understood. But I'm not at all convinced that the relational model is a good fit for a scale-out, widely distributed, fault-tolerant database. The engineering decisions for each of these aspects add enough latency to the process as it is.
- petermattis 10y ago
  (employee/founder here) Re joins: we have top men working on it right now. Joins are definitely an interesting problem for a distributed database.
  - chris_wot 10y ago
    Hi Peter, sounds pretty awesome! Quick question though - on your front page you say that CockroachDB does SQL - but if it can't do a join, then how can you say it uses SQL? Or is distributed SQL a different thing entirely? It does sounds like a very limited SQL subset though... I'm sure I must be missing something as I'm not familiar with your product. Also, what levels of isolation do you actually offer? Serialized snapshot isolation appears to be MVCC, but I see you also have just snapshot isolation - what is the difference? Edit: oh brother, the proof you link to, I just realized I bought that book some time ago and never got around to reading it... Transactional Information Systems by Weikum & Vossen, right? time for me to hit the books I guess. Still trying to get my head around that second graph you have drawn, can't work out how you have gotten it :(
    - knz42 10y ago
      SQL is not a single language but rather a family of languages. Each RDBMS that advertises support for SQL ends up implementing a different flavor of SQL, most of the times they are not even compatible :) For now CockroachDB's supports a subset of the SQL implemented by other databases, with some extensions of its own. This subset will grow over time. The two levels of isolation offered are snapshot (SI) and serializable (SSI). Snapshot means that concurrent transactions are atomic with regards to each other and "see" the same initial state from the DB. Serializable adds into this that they can't introduce write skews, ie if there are concurrent transactions they will "see" each others effects in some order.
    - petermattis 10y ago
      Hi Chris, we describe CockroachDB as a SQL database because that is what we're aspiring to. The missing functionality (i.e. joins) is on our near-term roadmap.
- jandrewrogers 10y ago
  It is possible to massively distribute joins but it requires a much more sophisticated database design than you see in most distributed database implementations because the whole system has to be designed for that use case e.g. support for ad hoc inter-node orchestration and data flows. At that point you basically have a parallel database instead of just a distributed one. And if you add an additional requirement to correctly execute ad hoc consistent joins under very high write workloads and continuous failures, then it is definitely non-trivial engineering. But it is possible, I've designed implementations like this before, it just requires a very high degree of effort and skill that is rarely applied to the design of typical NoSQL databases.
- manigandham 10y ago
  MemSQL does distributed joins really well, using a combination of in-memory rowstore + disk columnstore and the ability to have certain tables replicated to every node for faster joining.

danieltoshea 10y ago

How does this work if the clocks drift between the nodes? Does this allow incorrect behavior because one transaction looks like it happened before another?

knz42 10y ago
This was discussed earlier on the same blog: https://www.cockroachlabs.com/blog/living-without-atomic-clocks/ https://www.cockroachlabs.com/blog/living-without-atomic-clo... Short answer: the DB will check for drifts and correct small drifts automatically.
mrtracy 10y ago
(blog author here) Interestingly, clock drift does not affect the serializability of the transaction history; this system guarantees that the history is serializable, regardless of clock drift. However, "serializable" only means that the history is equivalent to some serial ordering of transactions - it makes no guarantee that the equivalent serial ordering is consistent with the real-time ordering of the involved transactions. A history with that property (agrees with real-time) is termed "linearizable", and requires additional rules to guarantee in an environment with clock drift. As mentioned by knz42, there was another Cockroach Labs blog post (written by Spencer Kimball) that addressed this in some detail; that blog post contrasted our strategy for dealing with drift with that of Google's Spanner. A quick overview of CockroachDB's properties re linearizability: it guarantees that access to any individual key is linearizable, and by composition any two transactions which share a key (that one of the transactions modifies) will be linearizable with respect to each other. However, if two transactions do not have any overlap in modified keys, Cockroach does not (by default) guarantee the resulting commit history is linearizable. CockroachDB's underlying KV layer does have a "linearizable" flag on transactions that can guarantee this, but it requires that transactions be slowed down considerably; Spencer's blog post addresses some other strategies that CockroachDB is considering to address the issue.
- lobster_johnson 10y ago
  Can Cockroach do the equivalent of a "select ... for update" (e.g., PostgreSQL), where you lock one thing while applying changes elsewhere? Concrete example: We have app that has a "documents" table and a "translog" table. The translog is like a series of diff-patches, representing changes to the documents. When we write to the translog, we first lock the document with a "select ... for update", so that no intervening translog entries can be written concurrently against the same document, then we patch the document, and then we write the translog entry and commit. We do this with Postgres, and we can do the same thing with Redis' MULTI since Redis is completely single-threaded. I can't think of any other NoSQL data store that allows a similar "lock A, update A, insert B, unlock A"; for example, Cassandra's "lightweight transactions" are only transactional in the context of a single row. (By "lock" I'd also accept optimistic locking, where you can retry on failure.)

barking 10y ago

Sorry if this is a little bit off topic This is a startup with employees that is making a non-commercial open source database system. How will they make a financial return on such a product?

sgarman 10y ago
It's common in the industry to build an open source product and then do paid support, hosting, etc with it. Joyent/Node Sonatype Nexus
- barking 10y ago
  OK thanks, I sort of thought that it might be something like that but wondered if I was missing some other angle.

LoSboccacc 10y ago

By coincidence I was looking around for a datastore. I almost settled on couchbase when I saw this; easy cluster deployment is my main draw after looking at how bad setting up master-master system is in other solutions (hbase, mongo, couchdb all have eterogeneus nodes and weird failure modes) One quick question since I saw the devs around and I can't find a final answer on it on google: are there any strong roadblock or performance drawbacks against storing principally medium size binary data like images, say, < 100mb blobs? Yeah I could use something more appropriate but I'd be back to figuring out hadoop installation or fighting s3 eventual consistency.

ansible 10y ago
I don't know that you would be happy trying to store blobs of that size in any distributed database, current or future. I'd suggest having an immutable distributed blob store, and just store the index in the distributed database. Of course, now you are half way towards a distributed filesystem...
- LoSboccacc 10y ago
  but then you'd need to store file path in the datastore, with all the failure modes that follows. I know what the current best practices are, but for bootstrapped one man startups you always end up with dozen moving parts to be tamed and instead of building stuff you need to solve the same problems over and over again I'm implementing a datastore on top of glusterfs, which while amateurish is a wonderful learning experience, but since it's nowhere close to being workable I am also looking around for what's happening in this space. I've this longer term goal about having a serverless framework where the datastore also provide oauth for users, which was inspired by https://github.com/serverless/serverless https://github.com/serverless/serverless - would be a boon for client side webapp app style startups.
  - manigandham 10y ago
    I have a strong sense that you're vastly over-engineering something. The solved problem is called an object store. Just use S3 or Azure Storage or Google Cloud Storage - they all have options for geo-replication and it'll be just as fast or faster than any database replication you set up, while being far more 9's of availability and reliability. Storing a tiny bit of text for the filename is trivial and you can use any durable database for that.

grogers 10y ago

Naively it looks like your conflict resolution is prone to livelock, where concurrent transactions for a key keep aborting each other without getting any work done. The same can happen if you run a single node RDBMS in serializable isolation mode and read a row before writing to it. In that case you can add locking reads on rows you intend to later write to, which avoids the livelock. Have you done any studies to quantify how bad that effect is in cockroachdb? Assuming the effect exists and I didn't just miss something silly, are there any workarounds since cockroachdb is only OCC? (Exponential backoff between retries and keeping transactions short don't count, I assume those are a given)

knz42 10y ago
When transactions conflict, the priority of the losing transaction is internally ratcheted up. At some point it's just higher than everything after it and it succeeeds.
mrtracy 10y ago
(author) This is a good insight; Live Lock was one of the things that kept me up at night, although my fellow contributors have allayed my concerns considerably. In a write-heavy workload where the transactions follow a read-modify-write pattern, it does seem possible that your application could be facing live lock concerns. So, let's look at where those concerns come from in CockroachDB: 1. For write-write conflicts, live-lock concerns comes from the fact that a later transaction can abort your in-progress transaction if it has a priority. On top of this, if your transaction ends up aborting an earlier transaction, the other transaction may retry with a potentially higher priority, and this can result in a "priority war" of sorts. 2. "Read-write" conflicts have a worse problem: if a transaction with a later timestamp reads a key before your transaction writes to it, then your transaction always aborts. In read-modify-write workloads where the read is first, this seems to be the biggest theoretical source of live-lock. Now, as for dealing with those issues. + The "Write-write" issue is the smaller concern here. The ratcheting-up of priorities is probabilistic; currently, after enough retries your transaction will have a high enough priority that it will almost certainly make progress. However, we are not totally content with this: we have an issue for our 1.0 release (https://github.com/cockroachdb/cockroach/issues/5727 https://github.com/cockroachdb/cockroach/issues/5727) to do some serious investigation of this, with some alternatives already suggested. For example, one such alternative is using "lowest original timestamp wins" to settle conflicts instead of a random priority. + The read-write concern seems more problematic; our read timestamp cache retains minimal data due to memory concerns, and thus aborts a bit conservatively (i.e. aborts transactions that might have been able to continue). This can be dealt with by modifying your application pattern - in particular, I would suggest using our "SNAPSHOT" isolation level for many workloads. I didn't cover this in the blog post, but in short the SNAPSHOT mode allows RW conflicts to occur without aborting transactions. SNAPSHOT transactions are subject to the "write skew" anomaly, but this anomaly does not occur if the involved transactions write to a common key. This is fortunately the case for many OLTP-type workloads, meaning SNAPSHOT can be used without anomalies; this also moves conflict-detection responsibility entirely to WW conflicts, which abort less conservatively than RW and should ameliorate any live-lock problems.

ruggeri 10y ago

I just wanted to thank the team for this writeup. I've been excited to read it since the last post several months back. This was definitely very useful, and very clearly written. So many thanks!