6 ms·

From RethinkDB docks [1], I am still a bit confused how this locking system works for read/write and also a bit skeptical regarding their claim that 'in most ca

From RethinkDB docks [1], I am still a bit confused how this locking system works for read/write and also a bit skeptical regarding their claim that 'in most cases writes can be performed essentially lock-free'. I am using MongoDB and didn't have many issues when my databases had 120,000 documents either, the problem began when we hit the millions... The combination of write locks and our need for dynamic queries (meaning: we can't index) made the database the worst performance bottleneck in our system by far. Although I must be honest that we haven't yet tried MongoDB's new 3.0 version that promises a boost in performance [2] and also has 'document-level locking and compression' [3] Is anybody aware of any benchmark that perform random writes (inserts/updates) and non-indexed reads for RethinkDB? (Is it even a common use scenario, anyways?) [1] http://rethinkdb.com/docs/architecture/#how-does-rethinkdb-execute-queries http://rethinkdb.com/docs/architecture/#how-does-rethinkdb-e... [2] http://www.mongodb.com/mongodb-3.0#performance http://www.mongodb.com/mongodb-3.0#performance [3] http://docs.mongodb.org/manual/release-notes/3.0/#wiredtiger-concurrency-and-compression http://docs.mongodb.org/manual/release-notes/3.0/#wiredtiger...

coffeemug 12y ago
> I am still a bit confused how this locking system works for read/write and also a bit skeptical regarding their claim that 'in most cases writes can be performed essentially lock-free'. Hi, Slava at RethinkDB here. RethinkDB uses MVCC to do looking. Essentially, when we lock down a block for a write, we make a copy of the block. If another query comes along that wants to read, it reads from the copy. When the write completes, the old copies are destroyed. There are lots of details I'm glossing over -- optimizations to avoid copying too much, copying entire subbranches of the btree to have a consistent view of the shard, etc. All this stuff isn't unique to RethinkDB -- it's pretty standard database internals stuff, and we haven't done anything new in that department. It's just an implementation of standard database architectures (as far as the caching/query engine/storage engine are concerned).
tracker1 12y ago
FYI, with MongoDB, just because you can't and shouldn't index everything, doesn't mean you can't have any indexes... if your most common fields bring your queries down, they're still pretty helpful. I actually really like where RethinkDB is headed, and within the year most of my issues should be resolved. Another couple databases to consider, depending on your needs would be ElasticSearch and Cassandra... it reallly depends on your use case.
- coffeemug 12y ago
  Could you list your issues with Rethink? It would really help for product prioritization.
  - tracker1 12y ago
    As mentioned in another thread, automagic failover the one still pending, and geospatial indexes/searches (now in the product). It's wild how many options there are that tailor themselves to all kinds of data out there.
    - pests 12y ago
      I'm not involved with RethinkDB but I lurk on their github issues and I'm pretty confident that automagic failover is dependent on them getting (their own implementation of) Raft integrated with everything. Looks like its getting close as a whole slew of issues were opened just the other day relating to Raft work.
    - deleted 12y ago
      [deleted]
  - jefficient 12y ago
    Are unique indexes planned for any of the upcoming releases?
    - coffeemug 12y ago
      The primary index for the table is always unique. We do not plan to implement secondary unique indexes. The philosophy behind RethinkDB is that if a feature cannot be efficiently scaled across multiple nodes we don't add it, and unfortunately unique secondary indexes are one such feature. AFAIK most NoSQL databases don't implement it, and a few that do take two approaches -- forbid it on multiple shards, or just take a massive performance hit during sharding. We chose to keep the feature out of the database. This way the application can be architected to account for it, so it remains fast as the database scales up.
lucian1900 12y ago
No database can be fast without indexes. If the queries you get are defined by your users, you can still create indexes for the common ones.
- felipesabino 12y ago
  The way business logic in the system is currently designed , there are no 'common ones', except for some ids (already indexed) used in other process rather than this dynamic filter. Based on that we also evaluated the approach of using something else like druid [1] [2] that is built for reading performance, but I am still studying possibilities and have no idea about the impact and problems a change like that would impose. [1] http://druid.io/ http://druid.io/ [2] https://metamarkets.com/2014/building-a-data-pipeline-that-handles-billions-of-events-in-real-time/ https://metamarkets.com/2014/building-a-data-pipeline-that-h...
  - lucian1900 12y ago
    Another thing to note is that Mongo is simply very slow. Without indexes, something like Postgres is much more likely to be fast.
deleted 12y ago
[deleted]
deleted 12y ago
[deleted]

coffeemug 12y ago

> I am still a bit confused how this locking system works for read/write and also a bit skeptical regarding their claim that 'in most cases writes can be performed essentially lock-free'. Hi, Slava at RethinkDB here. RethinkDB uses MVCC to do looking. Essentially, when we lock down a block for a write, we make a copy of the block. If another query comes along that wants to read, it reads from the copy. When the write completes, the old copies are destroyed. There are lots of details I'm glossing over -- optimizations to avoid copying too much, copying entire subbranches of the btree to have a consistent view of the shard, etc. All this stuff isn't unique to RethinkDB -- it's pretty standard database internals stuff, and we haven't done anything new in that department. It's just an implementation of standard database architectures (as far as the caching/query engine/storage engine are concerned).

tracker1 12y ago

FYI, with MongoDB, just because you can't and shouldn't index everything, doesn't mean you can't have any indexes... if your most common fields bring your queries down, they're still pretty helpful. I actually really like where RethinkDB is headed, and within the year most of my issues should be resolved. Another couple databases to consider, depending on your needs would be ElasticSearch and Cassandra... it reallly depends on your use case.

coffeemug 12y ago
Could you list your issues with Rethink? It would really help for product prioritization.
- tracker1 12y ago
  As mentioned in another thread, automagic failover the one still pending, and geospatial indexes/searches (now in the product). It's wild how many options there are that tailor themselves to all kinds of data out there.
  - pests 12y ago
    I'm not involved with RethinkDB but I lurk on their github issues and I'm pretty confident that automagic failover is dependent on them getting (their own implementation of) Raft integrated with everything. Looks like its getting close as a whole slew of issues were opened just the other day relating to Raft work.
  - deleted 12y ago
    [deleted]
- jefficient 12y ago
  Are unique indexes planned for any of the upcoming releases?
  - coffeemug 12y ago
    The primary index for the table is always unique. We do not plan to implement secondary unique indexes. The philosophy behind RethinkDB is that if a feature cannot be efficiently scaled across multiple nodes we don't add it, and unfortunately unique secondary indexes are one such feature. AFAIK most NoSQL databases don't implement it, and a few that do take two approaches -- forbid it on multiple shards, or just take a massive performance hit during sharding. We chose to keep the feature out of the database. This way the application can be architected to account for it, so it remains fast as the database scales up.

lucian1900 12y ago

No database can be fast without indexes. If the queries you get are defined by your users, you can still create indexes for the common ones.

felipesabino 12y ago
The way business logic in the system is currently designed , there are no 'common ones', except for some ids (already indexed) used in other process rather than this dynamic filter. Based on that we also evaluated the approach of using something else like druid [1] [2] that is built for reading performance, but I am still studying possibilities and have no idea about the impact and problems a change like that would impose. [1] http://druid.io/ http://druid.io/ [2] https://metamarkets.com/2014/building-a-data-pipeline-that-handles-billions-of-events-in-real-time/ https://metamarkets.com/2014/building-a-data-pipeline-that-h...
- lucian1900 12y ago
  Another thing to note is that Mongo is simply very slow. Without indexes, something like Postgres is much more likely to be fast.

deleted 12y ago

[deleted]

deleted 12y ago

[deleted]