3 ms·

Database consistency problems require traditional databases to use complicated locks. Because RethinkDB data is always consistent, locking is unnecessary. You

by neilc 17y ago

Database consistency problems require traditional databases to use complicated locks. Because RethinkDB data is always consistent, locking is unnecessary. You may not use locking for concurrency control (plenty of "traditional DBs" don't, either), but you still need some sort of concurrency control scheme -- just using append-only/log-structured storage doesn't make CC free. I'd be curious to hear how you guys are doing this.

coffeemug 17y ago
Most industrial strength storage engines use multiversion concurrency control, but they still require a fair share of locking - row level locks that protect the data while a second copy is made, for example. No storage engine is entirely "lock-free", but we have only very few extremely granular locks. This is completely different from what was done before (as far as we know), and from a practical perspective is as good as lock-free. I don't think I can explain the technology in a comment, but we'll definitely be releasing more information in the coming weeks. We had to get this release out for now. Stay tuned!
leif 17y ago
For the moment, we allow one writer at a time, but unbounded (I think) readers. The only locks taken are for arithmetic operations; no locks are held while performing I/O or even memcpy(2). You can read more about that bit in the paper about our caching architecture on the wiki. Eventually, we will allow multiple writers, and use some form of STM for that; we just haven't gotten to it yet. If you'd like to speak interactively on this, shoot me an email.
gaius 17y ago
Statements like that set my alarm bells ringing.
- neilc 17y ago
  Sorry, statements like what?
  - gaius 17y ago
    Statements like "traditional X does Y but..." The "traditional" (and I don't know how you could call the latest version of any of the major databases "traditional", this is a brutally competitive market) databases don't lock just for the fun of it, but to enable features that users want. Anyone can come up with a product that doesn't do Y if it can't do X either. So what're we missing here?
    - neilc 17y ago
      I don't know how you could call the latest version of any of the major databases "traditional" Oracle has done MVCC for many years, as has Postgres. The canonical paper on optimistic concurrency control for DBs is from 1981 (http://www.seas.upenn.edu/~zives/cis650/papers/opt-cc.pdf http://www.seas.upenn.edu/~zives/cis650/papers/opt-cc.pdf). I didn't follow the rest of your comment, I'm afraid -- I was just saying that I didn't see how using append-only storage immediately makes concurrency control a non-issue. The comments from the RethinkDB guys upthread support that: not supporting concurrent writers makes your concurrency control much more straightforward.

by neilc 17y ago

coffeemug 17y ago

Most industrial strength storage engines use multiversion concurrency control, but they still require a fair share of locking - row level locks that protect the data while a second copy is made, for example. No storage engine is entirely "lock-free", but we have only very few extremely granular locks. This is completely different from what was done before (as far as we know), and from a practical perspective is as good as lock-free. I don't think I can explain the technology in a comment, but we'll definitely be releasing more information in the coming weeks. We had to get this release out for now. Stay tuned!

leif 17y ago

For the moment, we allow one writer at a time, but unbounded (I think) readers. The only locks taken are for arithmetic operations; no locks are held while performing I/O or even memcpy(2). You can read more about that bit in the paper about our caching architecture on the wiki. Eventually, we will allow multiple writers, and use some form of STM for that; we just haven't gotten to it yet. If you'd like to speak interactively on this, shoot me an email.

gaius 17y ago

Statements like that set my alarm bells ringing.

neilc 17y ago
Sorry, statements like what?
- gaius 17y ago
  Statements like "traditional X does Y but..." The "traditional" (and I don't know how you could call the latest version of any of the major databases "traditional", this is a brutally competitive market) databases don't lock just for the fun of it, but to enable features that users want. Anyone can come up with a product that doesn't do Y if it can't do X either. So what're we missing here?
  - neilc 17y ago
    I don't know how you could call the latest version of any of the major databases "traditional" Oracle has done MVCC for many years, as has Postgres. The canonical paper on optimistic concurrency control for DBs is from 1981 (http://www.seas.upenn.edu/~zives/cis650/papers/opt-cc.pdf http://www.seas.upenn.edu/~zives/cis650/papers/opt-cc.pdf). I didn't follow the rest of your comment, I'm afraid -- I was just saying that I didn't see how using append-only storage immediately makes concurrency control a non-issue. The comments from the RethinkDB guys upthread support that: not supporting concurrent writers makes your concurrency control much more straightforward.