16 ms·

RethinkDB 2.0 is amazing

eternalban 11y ago
SQL is not "a set of arbitrary commands put together into a string". It is a bit ungainly and irregular in parts. It needs improvements[1]. Haskell (or F# or any FPL) are, of course and obviously, a perfect fit as front end for a relational system. [1]: http://www.thethirdmanifesto.com http://www.thethirdmanifesto.com
j4mie 11y ago
It's pretty amusing that "it writes your data to disk before acknowledging the write" is something that has to be described as "impressive" and written in bold text when talking about a database. MongoDB really has lowered the bar. That said, I love the look of Rethink and I can't wait to give it a try.
- dkhenry 11y ago
  Except it wasn't mongodb that did that. People had the same knock against MySQL with MyISAM.
  - army 11y ago
    Right, I'd be a lot more forgiving of MongoDB if they had been bringing the product to market 10-15 years earlier.
    - dkhenry 11y ago
      I am curious as to why. The underlying systems have only gotten more reliable and faster then they were 10-15 years ago. 10-15 years ago writing to disk was actually _more_ of a challenge then it is now with SSD's that have zero seek time.
      - zaphar 11y ago
        I don't think it's gotten any easier to verify that something was actually persisted to disk though. The hard part has always been verifying that the data is actually persisted to the hardware. And the number of layers between you and the physical storage has increased not decreased. And the number of those layers with a tendency to lie to you has increased not decreased. For some systems it's not considered to be persisted until it's been written to n+1 physical media for exactly these reasons. The os could be lying to you by buffering the write, the driver software for the disk could be lying to you as well by buffering the data. Even the physical hardware could be lying to you by buffering the write. In many ways writing may have gotten more reliable but verifying the write has gotten way harder.
    - jlarocco 11y ago
      Why? It was stupid and unsafe 10-15 years ago when MySQL was doing it, too, and all the devs who had been using more mature DBs (Oracle, DB2, etc.) complained about how bad it was.
      - DougMerritt 11y ago
        I was nodding in agreement right up until the word "Oracle". Essential any history of databases will say that for years, Oracle was not an RDBMS even by non-strict definitions (the claim is that Ellison didn't originally understand the concept correctly), and certainly did not offer ACID guarantees. Possibly Oracle had fixed 100% of that by the time MySQL came out, but now we're just talking about the timing of adding in safety, again -- and both IBM and Stonebraker's Ingres project (Postgres predecessor) had RDBMS with ACID in the late 1970s, and advertised the fact, so it wasn't a secret. Except in the early DOS/Windows world, where customers hadn't learned of the importance of reliability in hardware and software, and were more concerned simply with price. Oracle originally catered to that. MySQL did too, in some sense. In very recent years, it appears to me that people are re-learning the same lessons from scratch all over again, ignoring history, with certain kinds of recently popular databases.
- picardo 11y ago
  Writing to memory before writing to disk can be safe if you do it right. You need to deploy multiple instances in a replica set with a quorum threshold to guarantee safety. This is what Cassandra provides off the box. I don't think MongoDB made it clear at the start to its users that you should never work with a single database instance if you don't want to lose data.
  - yid 11y ago
    There's a very marked difference between "safe" and "low probability of failure". With uncommitted writes, even with a quorum, there's still a chance that you lose the write.
  - spotman 11y ago
    I have lost plenty of data on three separate occasions with mongo db, never running it by itself. always with at least a 3 member replica set. (this was 1-2 years ago, I'm sure it's improved). but it's not accurate to only blame the data loss issues on documentation.
- robconery 11y ago
  OP here - many NoSQL/document DBs will trade off write acks for eventual consistency. I really liked their approach to pushing toward durability by default - that in particular was the thing that impressed me, which I should have been more clear about.
  - picardo 11y ago
    Does it write to a disk or to a log? If it writes to disk, it may still not be completely fail-safe unless it also writes to a log before writing to disk. Postgres for example has a write ahead log and it writes to disk before acknowledging the write.
    - bmurphy1976 11y ago
      They use MVCC: http://rethinkdb.com/docs/architecture/ http://rethinkdb.com/docs/architecture/
      - eloff 11y ago
        So does postgres. That alone isn't enough because you can get situations like torn pages where part of the database page is old data and part is new data and nothing makes sense anymore. A log fixes that by first writing to the log so that if the database page gets messed up you have a secondary source you can use to restore it from.
    - coffeemug 11y ago
      RethinkDB has a log-structured storage engine. There is no separate log like in Postgres, the log is implicitly integrated into the storage engine. You don't have to write data twice (like you would with a traditional log), but you're still guaranteed safety in case of power failure. The design is roughly based on this paper: http://www.cs.berkeley.edu/~brewer/cs262/LFS.pdf http://www.cs.berkeley.edu/~brewer/cs262/LFS.pdf.
    - trhway 11y ago
      >Does it write to a disk or to a log? If it writes to disk, it may still not be completely fail-safe unless it also writes to a log before writing to disk. why we still discussing it at a tech forum in 21st century in Silicon Value? Shouldn't it (together with isolation, ACID, CAP, etc...) be a base knowledge taught in elementary school? Like you can't expect Daddy to buy you a firetruck that Mommy promised if Mommy hasn't been able to talk to Daddy yet... though until Mommy talks to Daddy you probably can convince Daddy to buy you a railroad...
- krakensden 11y ago
  For some reason I had it in my head that most databases don't actually block until fsync() returns- instead, the guarantee you get is that: # if execution continues, everything agrees on the state of the transaction # if execution halts, because of a crash or whatever, you'll come back online at a consistent state from the past
  - gaius 11y ago
    Typically when you COMMIT the changes will be written to the transaction log, which is sequential, then later written asynchronously to the data files. So you get the performance of sequential writes and the flexibility of random writes, which is nice. But once something is COMMIT'd it is permanent, it will survive any crash after COMMIT returns. If it has not yet be written to the datafiles, the recovery process will do that.
- istvan__ 11y ago
  Haha very true!! My #1 complain against MongoDB was the silent data loss scenario. Anyways I am curious what RethinkDB has to offer.
- reqres 11y ago
  There's a lot of FUD going around when it comes to MongoDB write durability. Please read the manual. Mongo lets the user decide whether or not to wait for fsync when writing to an individual node. This is not the default configuration. If you want it, you can enable it. You may complain that Mongo has bad defaults for your particular use case. It continues to have bad defaults to this day. Saying Mongodb is unable to acknowledge writes to disk is pure FUD. Let the downvotes ensue.
  - justicezyx 11y ago
    That's certainly a downgrade on the standard. Durability is a must not an option. The statement may not be friendly, but it is not down-vote-justifiable.
  - eloff 11y ago
    It's like if MySQL shipped with libeatmydata configured by default. The defaults should be safe. Mongo made not just a bad choice, but a really idiotic decision to make their default configuration non durable.
    - outworlder 11y ago
      Well, MySQL does ship like that. Only instead of not saving your data, it will mangle it in an effort to insert it in the database somehow.
    - omni 11y ago
      > The defaults should be safe. That's one opinion fitting one set of use cases. There are plenty of use cases where speed is more important than durability. Hell, Redis default configs don't enable the append-only log, but you don't see the HN hate train jumping all over Redis. This is because Redis use cases typically don't require that level of durability. edit for source: cmd+f for "appendonly" https://raw.githubusercontent.com/antirez/redis/2.8/redis.conf https://raw.githubusercontent.com/antirez/redis/2.8/redis.co...
      - eloff 11y ago
- easytiger 11y ago
  Everything you need to understand about persisting data to a physical medium has be written by richard hipp et al. You can't cheat. You can easily write async write calls in your own app to a synchronised storage engine and not assume the cost. When you have to have written logs which are accessed by government agencies you need to think about the atomicity of what a transaction is in you business. Is it when it enters the building as an electrical signal or when you flip the bits on a spinning disk Methinks the world has forgotten that high throughput systems existed long before the web of recent years. Most of what the web world thinks is high throughput is hilariously slow. The ability to run up another instance to scale sideways has ruined people. It doesn't scale in a linear fashion.
orthecreedence 11y ago
I can echo this post. I've been using Rethink for a few years now on side projects (as well as maintaining a driver, even if I skip a few server versions here and there...but these guys move fast). The composable queries alone are enough to make any developer happy. You can pretty much treat your data as if it's in-memory because the drivers integrate so well with the language. The relational model works really well. Things I have not tried yet are clustering and the real-time support (still need to build this into the lisp driver) but I'm trying to slot some time to do this. One of the projects I'm working on (https://turtl.it https://turtl.it) is going through a nice upgrade to mobile right now, and this will include some server changes...I'm looking forward to implementing changefeeds to solidify the collaboration aspect. Overall I've been really impressed with Rethink over the years, and can't express how excited I am they hit production ready. On top of the DB being great, the team is really nice to work with. They are incredibly responsive on github and were really helpful when I was first starting to build out my driver. Great post, and congrats to the Rethink team!
vasquque 11y ago
You aren't try http://tarantool.org http://tarantool.org where you can do with data anything.
- mrits 11y ago
  I know you speak at least one more language than I do.
shockzzz 11y ago
"RethinkDB is interesting" - absolutely "RethinkDB is amazing" - TBD I don't even think Slava would call RethinkDB "amazing" yet. I have no idea how to make a database, but I know there's a lot of work - and even more trial and error - that goes into making one "amazing." This is certainly a big step for RethinkDB. But I'd be careful to put Petabytes of data across 200 nodes sharded 500 ways each.
Scarbutt 11y ago
I'm sure there are use cases where both RethinkDB and Postgresql can equally provide a solution to some problem, but is it fair to compare them so blindly where one has ACID guarantees and the other one does not?
- robconery 11y ago
  OP here - I wanted to offer a comparison of the SQL vs. the ReQL query. Indeed if ACID is something you need, then yes a horizontally-scaling DB is probably something that deserves longer thought. This is a broader discussion to be sure, and it's been had. Given that PG now supports jsonb, it does mean that yes, we get to have these discussions more.
rdudekul 11y ago
Here is OP's TekpubTV video on RethinkDB (2 years old): https://vimeo.com/60697270 https://vimeo.com/60697270
- junto 11y ago
  Rob Conery always blows my mind. I came across him years ago when he used to evangelise .net stuff for MS. He wrote a great ORM with dB migrations that was completely novel for me at the time. He seems to be a complete polyglot and seems to he able to switch between so many different platforms, languages and technologies without any problems. Much respect!
army 11y ago
'When you query with ReQL you tack on functions and “compose” your data, with SQL you tell the query engine the steps necessary (aka prescribe) to return your data:' ... what? ReQL and SQL are both declarative query languages: I don't really see the author is getting at. Is there an implication that SQL isn't declarative? The only real difference is that the API is based around chaining function calls rather than expressing what is needed as a string - there are many SQL query builder APIs that will let you build SQL queries by chaining together function calls.
- bayesianhorse 11y ago
  One real difference is that you get most of the benefits of an ORM framework right in the driver, without depending on other packages or frameworks. And the API is very consistent across different languages. The biggest problem with composing SQL strings is that you have to be very very careful about SQL injections, and if you deal with that in a slightly sophisticated, reusable manner you are half-way to an ORM already. As far as I can determine, the ReQL drivers make injection attacks very difficult.
  - kbenson 11y ago
    The biggest problem with a different query language for every project is that when they get around to implementing some of the more esoteric (but extremely useful) SQL features it may not match well with what they've designed so far, and it may be hard to implement for the devs, conceptualize for the users, or just be plain weirdly tacked on causing a cognitive mismatch in how it's used (some side channel, for instance). Using a query builder (or ORM) of some sort still allows the escape hatch of raw SQL to do those really crazy things that are sometimes needed for performance, or just because what you are trying to do is rather weird. SQL is a very mature language, it's unlikely you are going to run into something someone else hasn't before.
    - nickik 11y ago
      Datomic uses datalog but also exposes lower levels of the data accses api. This allows people with special needs to drop down and do there own thing, while others can use datalog. It seams like a good idea to allow diffrent layers of data access.
BringTheTanks 11y ago
> Not earth-shattering, but with ReQL all I need to do is attach a function Thing is, some SQL databases have columnar storage, and in there selecting everything, then filtering with an attached function would eliminate the performance benefit of not selecting all of the fields. This is why SELECT looks like it does. Not to mention it's much shorter than attaching a function for the purpose. The author also himself acknowledges that: > The downside is that your queries end up quite long and, for some, rather intimidating. Ok so they're "quite long" and have less potential for optimizing the performance of. Amazing? His example of creating specific indexes and views is also not new to SQL. > There are 3 official drivers: Python, Ruby and Node. Amazing? > The query itself didn’t change at all – I could copy and paste it right in. I had to wrap it with connection info and a run() function, but that’s it. So just like an SQL query, except I can connect to an SQL RDBMS from virtually any language I can think of, and not just a narrow selection of 3 script languages. I sympathize with author's excitement, but from all his examples SQL feels like it has quite an edge both in availability and in terms of design and fit for the domain than a bunch of JS functions composed together (as much as I like composing functions together in JS). I realize how much hard work the folks at RethinkDB have put into creating their product. But technology adoption is not driven by pity, it's driven by benefits. For a new type of DB to not be a flash in the pan it needs a lot more than being "stable and fast". It needs to offer significant additional benefits when compared to existing DBs. And I ain't seeing it.
- army 11y ago
  Do you know for sure that RethinkDB can't work out what columns are being filtered by a function? In the examples it would certainly be possible with some analysis.
  - BringTheTanks 11y ago
    It can't work them out because you compose the query in a third party scripting language. RethinkDB has no access to the structure of the source in order to analyze it statically and work out an optimal I/O read plan. It interacts with the language runtime by providing an API and receiving callbacks to the API from the runtime. SQL is parsed & analyzed statically at the server, a plan is created based on that analysis and executed. So with SQL it is possible to do so. With RethinkDB you compose your query in the script, basically, and all of the optimization opportunities end with the exposed API (no function source analysis). It's not impossible to redesign the API to provide or even mandate static details like requested fields to RethinkDB, and it has a bit of that, but it allows freely mixing in client-side logic and even OP is confused about what it means to have a client-side mapping function. If they would allow complex expressions to run on the server, it'd become quite verbose to compose that via an API in an introspective way, to the point it'd warrant a DSL in a string... and we're back to SQL again.
WorldWideWayne 11y ago
All of the ReQL examples are completely unreadable relative to SQL. Call it declarative, format it however you want - a purpose-built DSL like SQL is always going to be easier to grok than a Javascript-inspired functional language (for me at least).
- bmurphy1976 11y ago
  Depends on your perspective I guess. Having struggled with building easily composable SQL queries for almost 20 years now, I'll take the functional language approach thank you very much.
  - WorldWideWayne 11y ago
    Thank you, I updated my statement to reflect that. However, I am imagining a contest where SQL people write SQL and functional people write functional queries... I would bet money that SQL people could identify basic facts about SQL queries faster than functional people could identify the same facts about functional queries. Could be a fun programming game.
- untog 11y ago
  All of the ReQL examples are completely unreadable relative to SQL. Because SQL is very familiar and ReQL is not.
  - sanderjd 11y ago
    I think it's more because the javascript-with-callbacks is chatty and ugly. But you can also write it in a more stream-like style. I have no idea why so many examples out there use the nastier style. An example stolen from another commenter: r.table('users').filter(r.row('age').gt(30))
    - coffeemug 11y ago
      > I think it's more because the javascript-with-callbacks is chatty and ugly. We did the best we could with JavaScript. IMO the Ruby implementation of ReQL is dramatically more beautiful. Ruby's blocks fit so well that we don't even provide the `r.row` shortcut in ruby: r.table('users').filter{|row| row['age'] > 30}
nawariata 11y ago
Does Rethink store keys with every record or uses more efficient mechanism?
- mrits 11y ago
  While you could technically use JSON to have a more efficient storage I think it safe to assume they use Key/Value. http://rethinkdb.com/docs/comparison-tables/ http://rethinkdb.com/docs/comparison-tables/ I can't believe new databases are still using this model. Some of my data storage would be 90% keys and 10% data as JSON.
pluma 11y ago
The API syntax isn't that special. The ArangoDB Query Builder (aqb on npm) for example is even more straightforward (no functions needed): a.for("album").in("catalog").filter( a.eq("album.details.media_type_id", 2) ).return("album") Or in plain AQL (ArangoDB's query language): FOR album IN catalog FILTER album.details.media_type_id == 2 RETURN album The "map" in the second example is simpler, too: ….return({artist: 'album.vendor.name'}) Or in plain AQL: … RETURN {artist: album.vendor.name} Also, it doesn't really need drivers because the DB uses a REST API that works with any HTTP client. That said, the change feeds are pretty neat and RethinkDB is still a pretty exciting project to follow. (Full disclosure: I wrote the ArangoDB Query Builder without any prior exposure to ReQL, so I may be biased)
- coffeemug 11y ago
  You don't actually need to use functions in ReQL either (although you can). For example r.table('users').filter(function(row) { return row('age').gt(30); }) Could be expressed as: r.table('users').filter(r.row('age').gt(30)) That being said aqb looks pretty cool and quite similar to ReQL.
  - pluma 11y ago
    Neat. I actually prefer the "infix" style for operators (i.e. having the methods on the values instead of on the helper) and I'll see whether I can adjust AQB to support that.
  - pluma 11y ago
    I've implemented the infix/ReQL style operators and published them in the latest release: https://github.com/arangodb/aqbjs/blob/v1.10.0/README.md#aql-operations https://github.com/arangodb/aqbjs/blob/v1.10.0/README.md#aql...
- deleted 11y ago
  [deleted]
- mmgutz 11y ago
  Postgres 9.3+ is fairly straight-forward too. Here is go + github.com/mgutz/dat // one trip to database using subqueries and Postgres' JSON functions con.SelectDoc("id", "user_name", "avatar"). HasMany("recent_comments", `SELECT id, title FROM comments WHERE id = users.id LIMIT 10`). HasMany("recent_posts", `SELECT id, title FROM posts WHERE author_id = users.id LIMIT 10`). HasOne("account", `SELECT balance FROM accounts WHERE user_id = users.id`). From("users"). Where("id = $1", 4). QueryStruct(&obj) // obj must be agreeable with json.Unmarshal() results in { "id": 4, "user_name": "mario", "avatar": "https://imgur.com/a23x.jpg", "recent_comments": [{"id": 1, "title": "..."}], "recent_posts": [{"id": 1, "title": "..."}], "account": { "balance": 42.00 } }
DonnyV 11y ago
I was interested in RethinkDB when I read the 2.0 release. But after looking at that mess he had to do to write a simple query. No thanks, I'll stick with Mongodb. db.catalog.find({ 'details.media_type_id': 2 })
- coffeemug 11y ago
  In RethinkDB you'd express it like this: r.table('catalog').filter({ details: { media_type: 2}}) Or like this: r.table('catalog').filter(r.row('details')('media_type').eq(2)) For most queries MongoDB syntax and RethinkDB syntax are effectively interchangeable.
- SamReidHughes 11y ago
  What do you do when a document has a period in its key?
  - justrudd 11y ago
    Restrictions on Field Names Field names cannot contain dots (i.e. .) or null characters, and they must not start with a dollar sign http://docs.mongodb.org/manual/reference/limits/#Restrictions-on-Field-Names http://docs.mongodb.org/manual/reference/limits/#Restriction...
DennisP 11y ago
Realtime update notifications, official node.js API...sounds like it'd be perfect as a second back end for Meteor.
- natebrennand 11y ago
  There's been some talk of it but they haven't committed to anything. https://groups.google.com/forum/#!searchin/rethinkdb/meteor/rethinkdb/XnzavMQvkO0/QJrHEP1Za9gJ https://groups.google.com/forum/#!searchin/rethinkdb/meteor/...
abhididdigi 11y ago
Along with all of this, the team behind rethink DB is really very helpful. Team was always patient when I was new. They always helped. This is one the very few projects that I felt right at home. Rethink DB had(during the days I was trying to use IRC) atleast one core developer available on IRC every day from Monday to Friday. Congratulations team!
Ciantic 11y ago
I can't imagine using RethinkDB until the query language ensures type safety. The article touts about SQL being full of strings, but ironically there are fully typed query builders for SQL but not for RethinkDB. All DB tables has schema even though it's not listed anywhere, and schemaless databases can't have fully typed query language either.
- coffeemug 11y ago
  Many core RethinkDB developers are huge fans of type safety. Check out the RethinkDB Haskell driver (https://github.com/atnnn/haskell-rethinkdb https://github.com/atnnn/haskell-rethinkdb), and the .NET driver (https://github.com/mfenniak/rethinkdb-net https://github.com/mfenniak/rethinkdb-net). ReQL works really nicely with type safety (and will work even better when we let people declare optional schemas).
  - wereHamster 11y ago
    Check out my Haskell driver (https://github.com/wereHamster/rethinkdb-client-driver https://github.com/wereHamster/rethinkdb-client-driver). I think think the first paragraph of the readme describes the differences between my driver and atnnn's quite nicely: > It differs from the other driver (rethinkdb) in that it uses advanced Haskell magic to properly type the terms, queries and responses. For example the driver knows that this query returns a number, and tries to parse it as such: call2 (lift (+)) (lift 1) (lift 2) Here are a few more examples from my application: -- | The primary key in all our documents is the default "id". primaryKeyField :: Text primaryKeyField = "id" -- | Expression which represents the primary key field. primaryKeyFieldE :: Exp Text primaryKeyFieldE = lift primaryKeyField -- | Expression which represents the value of a field inside of an Object. objectFieldE :: (IsDatum a) => Text -> Exp Object -> Exp a objectFieldE field obj = GetField (lift field) obj -- | True if the object field matches the given value. objectFieldEqE :: (ToDatum a) => Text -> a -> Exp Object -> Exp Bool objectFieldEqE field value obj = Eq (objectFieldE field obj :: Exp Datum) (lift $ toDatum value) -- | True if the object's primary key matches the given string. primaryKeyEqE :: Text -> Exp Object -> Exp Bool primaryKeyEqE = objectFieldEqE primaryKeyField My driver doesn't include all commands of the query language, just those which I need in my product. And I haven't tested it with RethinkDB 2.0 yet.
dmgbrn 11y ago
MongoDB has just done so much to erode my trust in novel databases. My knee jerk reaction is always "NOPE stickin with Postgres!". So I'm going to hold off on checking this one out, even though it seems from the comments that it's avoided many of Mongo's horrible design flaws. Just my 2 cents of Mongo hate :-)
- james33 11y ago
  When did you last use Mongo? We've been using it in production for 3+ years, and while there were certainly some issues early on, we've had nothing but success with it (especially over the last few major versions).
  - takeda 11y ago
    You could have inconsistencies in your data[1] and not even realize it. [1] https://aphyr.com/posts/284-call-me-maybe-mongodb https://aphyr.com/posts/284-call-me-maybe-mongodb
    - tracker1 11y ago
      While the call-me-maybe series is definitely informative... it's worth noting that they've called out flaws in every distributed system they've tested against. What it comes down to is, are those flaws fatal in practice. The truth is, it depends.... If you lose a comment on a social media site, no big deal. If you lose part of a transaction for a multi million dollar stock trade, very big deal. No software system is perfect, but there are definitely practical balances to be made. Especially when you are beyond what a single database/server can offer in terms of write throughput. The fact is, when your traffic needs exceed what a single database can keep up with in terms of writes, you have to give up some level of reliability.
      - takeda 11y ago
        He wasn't able to find issues with Zookeeper and Postgres. Granted that you can only prove that the system is vulnerable and not the reverse, but if there is a vulnerability it is much harder to trigger it.
StevePerkins 11y ago
Interesting that Wikipedia has no entry for RethinkDB. I do see that RethinkDB has some "Overview" and "FAQ" links on its website. However, when I encounter a new technology, I like to read its Wikipedia entry first. Wikipedia is usually more impartial, informative, and actually makes it easier to get a high-level sense of a technology than the tech's own website in most cases. This has grown more and more true over the past five or so years, as even developer-facing websites have devolved into "startup-y" marketing nonsense. I wonder if there WAS a Wikipedia entry, but it's been deleted by some moderator with an axe to grind? I personally haven't contributed in years due to how unpleasant it is to add new content through all of the Wiki-lawyering. I've also noticed that 5 years ago, when you did a Google search you could rely on the Wikipedia entry being one of the top 2 or 3 results. Lately I see more and more instances where I have to scroll to the second or third pages of results to see a Wikipedia link. Anyhoo... apologies for the tangential aside. I'm just wondering whether the lack of a Wikipedia entry says more about RethinkDB or about Wikipedia?
- coffeemug 11y ago
  Slava @ Rethink here. There used to be a few Wikipedia articles that kept getting deleted. I don't really understand the Wikipedia guidelines on this, but I don't worry about it too much. As RethinkDB grows the article will get added back, and it will get harder to make an argument that it should be deleted.
  - tptacek 11y ago
    There should be zero trouble getting a RethinkDB article written now, because it takes all of 5 seconds with NEWS.GOOGLE.COM to find reliable sources to back a notability claim. I can't find evidence of a deleted Rethink article in Wikipedia, but didn't look hard.
    - pests 11y ago
      Its listed right on the article's page: https://en.wikipedia.org/wiki/RethinkDB https://en.wikipedia.org/wiki/RethinkDB Reason: https://en.wikipedia.org/wiki/Wikipedia:Criteria_for_speedy_deletion#G11 https://en.wikipedia.org/wiki/Wikipedia:Criteria_for_speedy_... (G11. Unambiguous advertising or promotion)
mosselman 11y ago
So I have a question for you all. But first a little background. I have been playing with PouchDB, an in-the-browser implementation of CouchDB that is mostly compatible with CouchDB. What I really like is its sync function, this sets up no-brainer practically real-time sync between my local (offline) PouchDB database. It is very cool! I am very impressed by RethinkDB's cluster management, etc, so I would like to explore it as an option, but is there an easy way to sync my offline (browser based) localstorage-like database to rethink and back again? PouchDB makes this dead easy.
- coffeemug 11y ago
  There isn't a good way to do that yet. We've been playing with some ideas, but offline sync like this is a surprisingly challenging problem -- it's easy to make something that works, but dramatically harder to make something that works at scale.
  - mosselman 11y ago
    I can imagine that. I have seen some sync implementations for Backbone.js for instance a while back, but nothing that really worked well. Again, I am very surprised of how well PouchDB performs in this. Additionally, you get all (most) the features of CouchDB in the browser, you can even use the design documents that you create on the server.
- realusername 11y ago
  Excuse my curiosity unrelated to the current topic but I'm about to deploy a small scale hybrid (desktop & web) application using PouchDB. I have strongly unreliable networks (3G networks with entire days where the app is offline). The nodes are owning their own chunk of data so there is no risk of conflicts whatsoever, the main goal is to sync as soon as they can. PouchDB/CouchDB seemed clearly the best fit for this kind of unusual application. Did you encounter any problem with it ? Or if you can share your opinion on this technology after using it.
  - mosselman 11y ago
zak_mc_kracken 11y ago
It's a bit worrisome that there are no official Java drivers and the only unofficial Java project has been abandoned by its only author because he no longer uses Rethink.
- danielmewes 11y ago
  Daniel @ RethinkDB here. We're going to ship an official Java driver very soon. We decided to focus on a small number of core drivers first while the protocol and query language were still undergoing rapid extension. Now that the protocol is stable, we're going to expand our official driver support step by step. The Java one will be first. You can follow the progress on https://github.com/rethinkdb/rethinkdb/issues/3930 https://github.com/rethinkdb/rethinkdb/issues/3930
  - e2e4 11y ago
    Thats great news. I love RethinkDB; but not having a Java driver made it a no-go for a couple of projects.
- habitue 11y ago
  Hey there, I am literally working on the official Java driver right now, fear not!
peterbe 11y ago
I added a RethinkDB benchmark when used with the Python Tornado web server: http://www.peterbe.com/plog/fastestdb http://www.peterbe.com/plog/fastestdb Scroll down for the update.
- danielmewes 11y ago
  Note that (according to the linked commits), RethinkDB is using what we call "hard" durability in this comparison. This is our default to ensure maximum data safety. Hard durability means that every individual write will wait for the data to be written to disk before the next one is run (in this benchmark, since it only does one at a time). I don't think any of the other databases in this test is using a similarly strict requirement, are they? You'd have to run with the currently commented line "rethinkdb.db('talks').table_create('talks', durability='soft').run(conn)" to get more comparable results. (Edit for clarification: `durability='soft'` is comparable to the `safe` flag in many of the MongoDB drivers. It means that the server will acknowledge each write when it has been applied, but not wait for disk writes to complete.)
dcre 11y ago
I imagine the One Direction reference will be lost on most readers.
jfroma 11y ago
There is some FUD in this blogpost as well as in the comments in this thread. I think the current status quo for databases is canned software. And this isn't necessarily bad because neither of the three databases mentioned hide their specs or default settings, the three have very good docs and community willing to help, in addition to companies giving commercial support. Whats your excuse to misuse these products? RethinkDB writes your data to disk before acknowledging the write but on the other hand can't elect a new primary in case of failure, two completely different features/limitations that might work for someone and not for other ones. Is that hard to understand? Did mongo documentation lie you at some point? Accept that you are "buying" a general purpose product, the designers thought that their users will need those features, deal with it. Otherwise build your own database, I know this might sound very hard but I guess in the future we will see smaller building blocks that let you build something that handle your needs like this: https://github.com/rvagg/node-levelup/wiki/Modules#plugins https://github.com/rvagg/node-levelup/wiki/Modules#plugins
- NhanH 11y ago
  I was just thinking about this issue in the last few days. I'm working on a side project, due to the nature of the data model, converting back and forth to fit into a relational database is kind of annoying, so I was looking around for other databases. Right now, the situation with database is that we have to convert our internal data structure into a representation that fit the data model of the database we're using (ie rows for relational, document/key for the NoSQL group). I can see the reason the data model has to be that way for scaling, distributed etc... But if I'm happy to scale my database up, and would prefer to have the database storing the data as close to the memory data structure as possible (similar to object databases -- albeit with a boarder definition of "object"), is there any database that could do that? Otherwise, is there any suggestion on how I could get started to build one?
  - mrkurt 11y ago
    Redis is good at storing various data structures. Sets, sorted sets, hashes, etc. As long as "scaling up" means adding memory it's good.
- 11y ago
hoodoof 11y ago
There a common form of such posts by developers: "I've found database X and I LOVE it, we're migrating all our systems." Then a year later "Things didn't work out, here's how we migrated from database X to database Y." Someone should make an index of such blog posts.
sanguy 11y ago
The concern I have is that RethinkDB announced their death and vanished for 9+ months leaving everyone in a lurch, before resurfacing. The way they abandoned their users was deplorable so why trust them again? If you are into S&M just buy Oracle licenses...
- antonmaju 11y ago
  May I know when that happened? I'm curious about using RethinkDB after reading Rob's post
- coffeemug 11y ago
  Slava @ Rethink here. I'm not entirely sure what you're referring to, but the closest I can think of is the move from the memcached interface in 1.1 to the ReQL interface in 1.2. If that's what you mean, I don't think it's fair to say we abandoned our users at all. The memcached interface had very few people using it (literally single digits). We tried really hard to make it work, but there just wasn't any demand, so we decided to add a full query language, clustering, and rebuild with the realtime architecture. We supported the binary for a while, and helped most of the users migrate from the memcached interface to ReQL (which was fairly easy). We also helped people migrate to other memcached alternatives if they chose to not to use ReQL. In almost all cases people could quite literally pick another compatible product without changing any of their code. We took our time, helped people migrate (either to the newer version of RethinkDB, or to other products), and integrated the original architecture and as much of the code as we could into the new and improved RethinkDB. All of this was completely free of charge. So respectfully, I really don't think you're being fair to us. I'm sorry if this inconvenienced your company, but given the dire circumstances at the time we really did the best we could (and arguably, much, much more than most companies do in those circumstances).
pbz 11y ago
What do you guys think of Aerospike (http://www.aerospike.com/ http://www.aerospike.com/)? I heard of a few cases that used it successfully. Any issues with it?
- chupy 11y ago
  Aerospike - distributed key value store has a little bit different of a use case than RethinkDB - distributed JSON document databases. Basically the only valid use that I have seen for Aerospike, and that is the one that they are advertising is a distributed key value profile store for Ad-tech or marketing companies (http://www.aerospike.com/overview/ http://www.aerospike.com/overview/). Keep in mind that to actually use all of Aerospike's features , especially the one they are really proud about - the cross datacenter replication - you need the commercial license. So..I suggest you figure out what your requirements are and then use the best tool for the job.
  - bbulkow 11y ago
    CTO at Aerospike here. Aerospike is battle-tested in large deployments --- ad-tech, marketing-tech, a few new ones in telecom and fin-serv. Pushing huge load with very, very little downtime. That's what we're the most proud of --- and I'm proud that we're able to offer this killer codebase as open source, after being closed source for the first few years of the company. Most applications have a huge core of key-value --- twitter, for example --- and need a fast and scalable key-value component. You can start with a single server (on your laptop with Vagrant) and scale up later. We're adding more types, more cool operations, more indexes this year. The fact that Aerospike has a basic query system, type safety, flash optimization (Amazon has switched over to being very SSD/Flash centric) support for every language under the sun (three contributed Scala layers --- and we see a lot of Go use as well as the usual Java / Node / Python / PHP / HHVM), Hadoop integration....
beefsack 11y ago
Have used RethinkDB with a few projects now, mainly in Go with dancannon's Go driver[1], and have found it very functional and robust and has become my go-to schemaless database. My largest project has been running a couple of years now and has accumulated a significant amount of data, and RethinkDB hasn't had any trouble at all scaling with my data growth. I'm running it on servers below the recommended requirements too (512MB DO instances) and have been really impressed with how it handles constrained resources. [1] https://github.com/dancannon/gorethink https://github.com/dancannon/gorethink
timmaxw 11y ago
Note that the 9ms performance claim in this article is mistaken. The author has issued a correction: http://rob.conery.io/2015/04/17/rethinkdb-2-0-is-amazing/ http://rob.conery.io/2015/04/17/rethinkdb-2-0-is-amazing/

SQL is not "a set of arbitrary commands put together into a string". It is a bit ungainly and irregular in parts. It needs improvements[1]. Haskell (or F# or any FPL) are, of course and obviously, a perfect fit as front end for a relational system. [1]: http://www.thethirdmanifesto.com http://www.thethirdmanifesto.com

j4mie 11y ago

It's pretty amusing that "it writes your data to disk before acknowledging the write" is something that has to be described as "impressive" and written in bold text when talking about a database. MongoDB really has lowered the bar. That said, I love the look of Rethink and I can't wait to give it a try.

dkhenry 11y ago
Except it wasn't mongodb that did that. People had the same knock against MySQL with MyISAM.
- army 11y ago
  Right, I'd be a lot more forgiving of MongoDB if they had been bringing the product to market 10-15 years earlier.
  - dkhenry 11y ago
    I am curious as to why. The underlying systems have only gotten more reliable and faster then they were 10-15 years ago. 10-15 years ago writing to disk was actually _more_ of a challenge then it is now with SSD's that have zero seek time.
    - zaphar 11y ago
      I don't think it's gotten any easier to verify that something was actually persisted to disk though. The hard part has always been verifying that the data is actually persisted to the hardware. And the number of layers between you and the physical storage has increased not decreased. And the number of those layers with a tendency to lie to you has increased not decreased. For some systems it's not considered to be persisted until it's been written to n+1 physical media for exactly these reasons. The os could be lying to you by buffering the write, the driver software for the disk could be lying to you as well by buffering the data. Even the physical hardware could be lying to you by buffering the write. In many ways writing may have gotten more reliable but verifying the write has gotten way harder.
  - jlarocco 11y ago
    Why? It was stupid and unsafe 10-15 years ago when MySQL was doing it, too, and all the devs who had been using more mature DBs (Oracle, DB2, etc.) complained about how bad it was.
    - DougMerritt 11y ago
      I was nodding in agreement right up until the word "Oracle". Essential any history of databases will say that for years, Oracle was not an RDBMS even by non-strict definitions (the claim is that Ellison didn't originally understand the concept correctly), and certainly did not offer ACID guarantees. Possibly Oracle had fixed 100% of that by the time MySQL came out, but now we're just talking about the timing of adding in safety, again -- and both IBM and Stonebraker's Ingres project (Postgres predecessor) had RDBMS with ACID in the late 1970s, and advertised the fact, so it wasn't a secret. Except in the early DOS/Windows world, where customers hadn't learned of the importance of reliability in hardware and software, and were more concerned simply with price. Oracle originally catered to that. MySQL did too, in some sense. In very recent years, it appears to me that people are re-learning the same lessons from scratch all over again, ignoring history, with certain kinds of recently popular databases.
picardo 11y ago
Writing to memory before writing to disk can be safe if you do it right. You need to deploy multiple instances in a replica set with a quorum threshold to guarantee safety. This is what Cassandra provides off the box. I don't think MongoDB made it clear at the start to its users that you should never work with a single database instance if you don't want to lose data.
- yid 11y ago
  There's a very marked difference between "safe" and "low probability of failure". With uncommitted writes, even with a quorum, there's still a chance that you lose the write.
- spotman 11y ago
  I have lost plenty of data on three separate occasions with mongo db, never running it by itself. always with at least a 3 member replica set. (this was 1-2 years ago, I'm sure it's improved). but it's not accurate to only blame the data loss issues on documentation.
robconery 11y ago
OP here - many NoSQL/document DBs will trade off write acks for eventual consistency. I really liked their approach to pushing toward durability by default - that in particular was the thing that impressed me, which I should have been more clear about.
- picardo 11y ago
  Does it write to a disk or to a log? If it writes to disk, it may still not be completely fail-safe unless it also writes to a log before writing to disk. Postgres for example has a write ahead log and it writes to disk before acknowledging the write.
  - bmurphy1976 11y ago
    They use MVCC: http://rethinkdb.com/docs/architecture/ http://rethinkdb.com/docs/architecture/
    - eloff 11y ago
      So does postgres. That alone isn't enough because you can get situations like torn pages where part of the database page is old data and part is new data and nothing makes sense anymore. A log fixes that by first writing to the log so that if the database page gets messed up you have a secondary source you can use to restore it from.
  - coffeemug 11y ago
    RethinkDB has a log-structured storage engine. There is no separate log like in Postgres, the log is implicitly integrated into the storage engine. You don't have to write data twice (like you would with a traditional log), but you're still guaranteed safety in case of power failure. The design is roughly based on this paper: http://www.cs.berkeley.edu/~brewer/cs262/LFS.pdf http://www.cs.berkeley.edu/~brewer/cs262/LFS.pdf.
  - trhway 11y ago
    >Does it write to a disk or to a log? If it writes to disk, it may still not be completely fail-safe unless it also writes to a log before writing to disk. why we still discussing it at a tech forum in 21st century in Silicon Value? Shouldn't it (together with isolation, ACID, CAP, etc...) be a base knowledge taught in elementary school? Like you can't expect Daddy to buy you a firetruck that Mommy promised if Mommy hasn't been able to talk to Daddy yet... though until Mommy talks to Daddy you probably can convince Daddy to buy you a railroad...
krakensden 11y ago
For some reason I had it in my head that most databases don't actually block until fsync() returns- instead, the guarantee you get is that: # if execution continues, everything agrees on the state of the transaction # if execution halts, because of a crash or whatever, you'll come back online at a consistent state from the past
- gaius 11y ago
  Typically when you COMMIT the changes will be written to the transaction log, which is sequential, then later written asynchronously to the data files. So you get the performance of sequential writes and the flexibility of random writes, which is nice. But once something is COMMIT'd it is permanent, it will survive any crash after COMMIT returns. If it has not yet be written to the datafiles, the recovery process will do that.
istvan__ 11y ago
Haha very true!! My #1 complain against MongoDB was the silent data loss scenario. Anyways I am curious what RethinkDB has to offer.
reqres 11y ago
There's a lot of FUD going around when it comes to MongoDB write durability. Please read the manual. Mongo lets the user decide whether or not to wait for fsync when writing to an individual node. This is not the default configuration. If you want it, you can enable it. You may complain that Mongo has bad defaults for your particular use case. It continues to have bad defaults to this day. Saying Mongodb is unable to acknowledge writes to disk is pure FUD. Let the downvotes ensue.
- justicezyx 11y ago
  That's certainly a downgrade on the standard. Durability is a must not an option. The statement may not be friendly, but it is not down-vote-justifiable.
- eloff 11y ago
  It's like if MySQL shipped with libeatmydata configured by default. The defaults should be safe. Mongo made not just a bad choice, but a really idiotic decision to make their default configuration non durable.
  - outworlder 11y ago
    Well, MySQL does ship like that. Only instead of not saving your data, it will mangle it in an effort to insert it in the database somehow.
  - omni 11y ago
    > The defaults should be safe. That's one opinion fitting one set of use cases. There are plenty of use cases where speed is more important than durability. Hell, Redis default configs don't enable the append-only log, but you don't see the HN hate train jumping all over Redis. This is because Redis use cases typically don't require that level of durability. edit for source: cmd+f for "appendonly" https://raw.githubusercontent.com/antirez/redis/2.8/redis.conf https://raw.githubusercontent.com/antirez/redis/2.8/redis.co...
    - eloff 11y ago
easytiger 11y ago
Everything you need to understand about persisting data to a physical medium has be written by richard hipp et al. You can't cheat. You can easily write async write calls in your own app to a synchronised storage engine and not assume the cost. When you have to have written logs which are accessed by government agencies you need to think about the atomicity of what a transaction is in you business. Is it when it enters the building as an electrical signal or when you flip the bits on a spinning disk Methinks the world has forgotten that high throughput systems existed long before the web of recent years. Most of what the web world thinks is high throughput is hilariously slow. The ability to run up another instance to scale sideways has ruined people. It doesn't scale in a linear fashion.

orthecreedence 11y ago

I can echo this post. I've been using Rethink for a few years now on side projects (as well as maintaining a driver, even if I skip a few server versions here and there...but these guys move fast). The composable queries alone are enough to make any developer happy. You can pretty much treat your data as if it's in-memory because the drivers integrate so well with the language. The relational model works really well. Things I have not tried yet are clustering and the real-time support (still need to build this into the lisp driver) but I'm trying to slot some time to do this. One of the projects I'm working on (https://turtl.it https://turtl.it) is going through a nice upgrade to mobile right now, and this will include some server changes...I'm looking forward to implementing changefeeds to solidify the collaboration aspect. Overall I've been really impressed with Rethink over the years, and can't express how excited I am they hit production ready. On top of the DB being great, the team is really nice to work with. They are incredibly responsive on github and were really helpful when I was first starting to build out my driver. Great post, and congrats to the Rethink team!

vasquque 11y ago

You aren't try http://tarantool.org http://tarantool.org where you can do with data anything.

mrits 11y ago
I know you speak at least one more language than I do.

shockzzz 11y ago

"RethinkDB is interesting" - absolutely "RethinkDB is amazing" - TBD I don't even think Slava would call RethinkDB "amazing" yet. I have no idea how to make a database, but I know there's a lot of work - and even more trial and error - that goes into making one "amazing." This is certainly a big step for RethinkDB. But I'd be careful to put Petabytes of data across 200 nodes sharded 500 ways each.

Scarbutt 11y ago

I'm sure there are use cases where both RethinkDB and Postgresql can equally provide a solution to some problem, but is it fair to compare them so blindly where one has ACID guarantees and the other one does not?

robconery 11y ago
OP here - I wanted to offer a comparison of the SQL vs. the ReQL query. Indeed if ACID is something you need, then yes a horizontally-scaling DB is probably something that deserves longer thought. This is a broader discussion to be sure, and it's been had. Given that PG now supports jsonb, it does mean that yes, we get to have these discussions more.

rdudekul 11y ago

Here is OP's TekpubTV video on RethinkDB (2 years old): https://vimeo.com/60697270 https://vimeo.com/60697270

junto 11y ago
Rob Conery always blows my mind. I came across him years ago when he used to evangelise .net stuff for MS. He wrote a great ORM with dB migrations that was completely novel for me at the time. He seems to be a complete polyglot and seems to he able to switch between so many different platforms, languages and technologies without any problems. Much respect!

army 11y ago

'When you query with ReQL you tack on functions and “compose” your data, with SQL you tell the query engine the steps necessary (aka prescribe) to return your data:' ... what? ReQL and SQL are both declarative query languages: I don't really see the author is getting at. Is there an implication that SQL isn't declarative? The only real difference is that the API is based around chaining function calls rather than expressing what is needed as a string - there are many SQL query builder APIs that will let you build SQL queries by chaining together function calls.

bayesianhorse 11y ago
One real difference is that you get most of the benefits of an ORM framework right in the driver, without depending on other packages or frameworks. And the API is very consistent across different languages. The biggest problem with composing SQL strings is that you have to be very very careful about SQL injections, and if you deal with that in a slightly sophisticated, reusable manner you are half-way to an ORM already. As far as I can determine, the ReQL drivers make injection attacks very difficult.
- kbenson 11y ago
  The biggest problem with a different query language for every project is that when they get around to implementing some of the more esoteric (but extremely useful) SQL features it may not match well with what they've designed so far, and it may be hard to implement for the devs, conceptualize for the users, or just be plain weirdly tacked on causing a cognitive mismatch in how it's used (some side channel, for instance). Using a query builder (or ORM) of some sort still allows the escape hatch of raw SQL to do those really crazy things that are sometimes needed for performance, or just because what you are trying to do is rather weird. SQL is a very mature language, it's unlikely you are going to run into something someone else hasn't before.
  - nickik 11y ago
    Datomic uses datalog but also exposes lower levels of the data accses api. This allows people with special needs to drop down and do there own thing, while others can use datalog. It seams like a good idea to allow diffrent layers of data access.

BringTheTanks 11y ago

> Not earth-shattering, but with ReQL all I need to do is attach a function Thing is, some SQL databases have columnar storage, and in there selecting everything, then filtering with an attached function would eliminate the performance benefit of not selecting all of the fields. This is why SELECT looks like it does. Not to mention it's much shorter than attaching a function for the purpose. The author also himself acknowledges that: > The downside is that your queries end up quite long and, for some, rather intimidating. Ok so they're "quite long" and have less potential for optimizing the performance of. Amazing? His example of creating specific indexes and views is also not new to SQL. > There are 3 official drivers: Python, Ruby and Node. Amazing? > The query itself didn’t change at all – I could copy and paste it right in. I had to wrap it with connection info and a run() function, but that’s it. So just like an SQL query, except I can connect to an SQL RDBMS from virtually any language I can think of, and not just a narrow selection of 3 script languages. I sympathize with author's excitement, but from all his examples SQL feels like it has quite an edge both in availability and in terms of design and fit for the domain than a bunch of JS functions composed together (as much as I like composing functions together in JS). I realize how much hard work the folks at RethinkDB have put into creating their product. But technology adoption is not driven by pity, it's driven by benefits. For a new type of DB to not be a flash in the pan it needs a lot more than being "stable and fast". It needs to offer significant additional benefits when compared to existing DBs. And I ain't seeing it.

army 11y ago
Do you know for sure that RethinkDB can't work out what columns are being filtered by a function? In the examples it would certainly be possible with some analysis.
- BringTheTanks 11y ago
  It can't work them out because you compose the query in a third party scripting language. RethinkDB has no access to the structure of the source in order to analyze it statically and work out an optimal I/O read plan. It interacts with the language runtime by providing an API and receiving callbacks to the API from the runtime. SQL is parsed & analyzed statically at the server, a plan is created based on that analysis and executed. So with SQL it is possible to do so. With RethinkDB you compose your query in the script, basically, and all of the optimization opportunities end with the exposed API (no function source analysis). It's not impossible to redesign the API to provide or even mandate static details like requested fields to RethinkDB, and it has a bit of that, but it allows freely mixing in client-side logic and even OP is confused about what it means to have a client-side mapping function. If they would allow complex expressions to run on the server, it'd become quite verbose to compose that via an API in an introspective way, to the point it'd warrant a DSL in a string... and we're back to SQL again.

WorldWideWayne 11y ago

All of the ReQL examples are completely unreadable relative to SQL. Call it declarative, format it however you want - a purpose-built DSL like SQL is always going to be easier to grok than a Javascript-inspired functional language (for me at least).

bmurphy1976 11y ago
Depends on your perspective I guess. Having struggled with building easily composable SQL queries for almost 20 years now, I'll take the functional language approach thank you very much.
- WorldWideWayne 11y ago
  Thank you, I updated my statement to reflect that. However, I am imagining a contest where SQL people write SQL and functional people write functional queries... I would bet money that SQL people could identify basic facts about SQL queries faster than functional people could identify the same facts about functional queries. Could be a fun programming game.
untog 11y ago
All of the ReQL examples are completely unreadable relative to SQL. Because SQL is very familiar and ReQL is not.
- sanderjd 11y ago
  I think it's more because the javascript-with-callbacks is chatty and ugly. But you can also write it in a more stream-like style. I have no idea why so many examples out there use the nastier style. An example stolen from another commenter: r.table('users').filter(r.row('age').gt(30))
  - coffeemug 11y ago
    > I think it's more because the javascript-with-callbacks is chatty and ugly. We did the best we could with JavaScript. IMO the Ruby implementation of ReQL is dramatically more beautiful. Ruby's blocks fit so well that we don't even provide the `r.row` shortcut in ruby: r.table('users').filter{|row| row['age'] > 30}

nawariata 11y ago

Does Rethink store keys with every record or uses more efficient mechanism?

mrits 11y ago
While you could technically use JSON to have a more efficient storage I think it safe to assume they use Key/Value. http://rethinkdb.com/docs/comparison-tables/ http://rethinkdb.com/docs/comparison-tables/ I can't believe new databases are still using this model. Some of my data storage would be 90% keys and 10% data as JSON.

pluma 11y ago

The API syntax isn't that special. The ArangoDB Query Builder (aqb on npm) for example is even more straightforward (no functions needed): a.for("album").in("catalog").filter( a.eq("album.details.media_type_id", 2) ).return("album") Or in plain AQL (ArangoDB's query language): FOR album IN catalog FILTER album.details.media_type_id == 2 RETURN album The "map" in the second example is simpler, too: ….return({artist: 'album.vendor.name'}) Or in plain AQL: … RETURN {artist: album.vendor.name} Also, it doesn't really need drivers because the DB uses a REST API that works with any HTTP client. That said, the change feeds are pretty neat and RethinkDB is still a pretty exciting project to follow. (Full disclosure: I wrote the ArangoDB Query Builder without any prior exposure to ReQL, so I may be biased)

coffeemug 11y ago
You don't actually need to use functions in ReQL either (although you can). For example r.table('users').filter(function(row) { return row('age').gt(30); }) Could be expressed as: r.table('users').filter(r.row('age').gt(30)) That being said aqb looks pretty cool and quite similar to ReQL.
- pluma 11y ago
  Neat. I actually prefer the "infix" style for operators (i.e. having the methods on the values instead of on the helper) and I'll see whether I can adjust AQB to support that.
- pluma 11y ago
  I've implemented the infix/ReQL style operators and published them in the latest release: https://github.com/arangodb/aqbjs/blob/v1.10.0/README.md#aql-operations https://github.com/arangodb/aqbjs/blob/v1.10.0/README.md#aql...
deleted 11y ago
[deleted]
mmgutz 11y ago
Postgres 9.3+ is fairly straight-forward too. Here is go + github.com/mgutz/dat // one trip to database using subqueries and Postgres' JSON functions con.SelectDoc("id", "user_name", "avatar"). HasMany("recent_comments", `SELECT id, title FROM comments WHERE id = users.id LIMIT 10`). HasMany("recent_posts", `SELECT id, title FROM posts WHERE author_id = users.id LIMIT 10`). HasOne("account", `SELECT balance FROM accounts WHERE user_id = users.id`). From("users"). Where("id = $1", 4). QueryStruct(&obj) // obj must be agreeable with json.Unmarshal() results in { "id": 4, "user_name": "mario", "avatar": "https://imgur.com/a23x.jpg", "recent_comments": [{"id": 1, "title": "..."}], "recent_posts": [{"id": 1, "title": "..."}], "account": { "balance": 42.00 } }

DonnyV 11y ago

I was interested in RethinkDB when I read the 2.0 release. But after looking at that mess he had to do to write a simple query. No thanks, I'll stick with Mongodb. db.catalog.find({ 'details.media_type_id': 2 })

coffeemug 11y ago
In RethinkDB you'd express it like this: r.table('catalog').filter({ details: { media_type: 2}}) Or like this: r.table('catalog').filter(r.row('details')('media_type').eq(2)) For most queries MongoDB syntax and RethinkDB syntax are effectively interchangeable.
SamReidHughes 11y ago
What do you do when a document has a period in its key?
- justrudd 11y ago
  Restrictions on Field Names Field names cannot contain dots (i.e. .) or null characters, and they must not start with a dollar sign http://docs.mongodb.org/manual/reference/limits/#Restrictions-on-Field-Names http://docs.mongodb.org/manual/reference/limits/#Restriction...

DennisP 11y ago

Realtime update notifications, official node.js API...sounds like it'd be perfect as a second back end for Meteor.

natebrennand 11y ago
There's been some talk of it but they haven't committed to anything. https://groups.google.com/forum/#!searchin/rethinkdb/meteor/rethinkdb/XnzavMQvkO0/QJrHEP1Za9gJ https://groups.google.com/forum/#!searchin/rethinkdb/meteor/...

abhididdigi 11y ago

Along with all of this, the team behind rethink DB is really very helpful. Team was always patient when I was new. They always helped. This is one the very few projects that I felt right at home. Rethink DB had(during the days I was trying to use IRC) atleast one core developer available on IRC every day from Monday to Friday. Congratulations team!

Ciantic 11y ago

I can't imagine using RethinkDB until the query language ensures type safety. The article touts about SQL being full of strings, but ironically there are fully typed query builders for SQL but not for RethinkDB. All DB tables has schema even though it's not listed anywhere, and schemaless databases can't have fully typed query language either.

coffeemug 11y ago
Many core RethinkDB developers are huge fans of type safety. Check out the RethinkDB Haskell driver (https://github.com/atnnn/haskell-rethinkdb https://github.com/atnnn/haskell-rethinkdb), and the .NET driver (https://github.com/mfenniak/rethinkdb-net https://github.com/mfenniak/rethinkdb-net). ReQL works really nicely with type safety (and will work even better when we let people declare optional schemas).
- wereHamster 11y ago
  Check out my Haskell driver (https://github.com/wereHamster/rethinkdb-client-driver https://github.com/wereHamster/rethinkdb-client-driver). I think think the first paragraph of the readme describes the differences between my driver and atnnn's quite nicely: > It differs from the other driver (rethinkdb) in that it uses advanced Haskell magic to properly type the terms, queries and responses. For example the driver knows that this query returns a number, and tries to parse it as such: call2 (lift (+)) (lift 1) (lift 2) Here are a few more examples from my application: -- | The primary key in all our documents is the default "id". primaryKeyField :: Text primaryKeyField = "id" -- | Expression which represents the primary key field. primaryKeyFieldE :: Exp Text primaryKeyFieldE = lift primaryKeyField -- | Expression which represents the value of a field inside of an Object. objectFieldE :: (IsDatum a) => Text -> Exp Object -> Exp a objectFieldE field obj = GetField (lift field) obj -- | True if the object field matches the given value. objectFieldEqE :: (ToDatum a) => Text -> a -> Exp Object -> Exp Bool objectFieldEqE field value obj = Eq (objectFieldE field obj :: Exp Datum) (lift $ toDatum value) -- | True if the object's primary key matches the given string. primaryKeyEqE :: Text -> Exp Object -> Exp Bool primaryKeyEqE = objectFieldEqE primaryKeyField My driver doesn't include all commands of the query language, just those which I need in my product. And I haven't tested it with RethinkDB 2.0 yet.

dmgbrn 11y ago

MongoDB has just done so much to erode my trust in novel databases. My knee jerk reaction is always "NOPE stickin with Postgres!". So I'm going to hold off on checking this one out, even though it seems from the comments that it's avoided many of Mongo's horrible design flaws. Just my 2 cents of Mongo hate :-)

james33 11y ago
When did you last use Mongo? We've been using it in production for 3+ years, and while there were certainly some issues early on, we've had nothing but success with it (especially over the last few major versions).
- takeda 11y ago
  You could have inconsistencies in your data[1] and not even realize it. [1] https://aphyr.com/posts/284-call-me-maybe-mongodb https://aphyr.com/posts/284-call-me-maybe-mongodb
  - tracker1 11y ago
    While the call-me-maybe series is definitely informative... it's worth noting that they've called out flaws in every distributed system they've tested against. What it comes down to is, are those flaws fatal in practice. The truth is, it depends.... If you lose a comment on a social media site, no big deal. If you lose part of a transaction for a multi million dollar stock trade, very big deal. No software system is perfect, but there are definitely practical balances to be made. Especially when you are beyond what a single database/server can offer in terms of write throughput. The fact is, when your traffic needs exceed what a single database can keep up with in terms of writes, you have to give up some level of reliability.
    - takeda 11y ago
      He wasn't able to find issues with Zookeeper and Postgres. Granted that you can only prove that the system is vulnerable and not the reverse, but if there is a vulnerability it is much harder to trigger it.

StevePerkins 11y ago

Interesting that Wikipedia has no entry for RethinkDB. I do see that RethinkDB has some "Overview" and "FAQ" links on its website. However, when I encounter a new technology, I like to read its Wikipedia entry first. Wikipedia is usually more impartial, informative, and actually makes it easier to get a high-level sense of a technology than the tech's own website in most cases. This has grown more and more true over the past five or so years, as even developer-facing websites have devolved into "startup-y" marketing nonsense. I wonder if there WAS a Wikipedia entry, but it's been deleted by some moderator with an axe to grind? I personally haven't contributed in years due to how unpleasant it is to add new content through all of the Wiki-lawyering. I've also noticed that 5 years ago, when you did a Google search you could rely on the Wikipedia entry being one of the top 2 or 3 results. Lately I see more and more instances where I have to scroll to the second or third pages of results to see a Wikipedia link. Anyhoo... apologies for the tangential aside. I'm just wondering whether the lack of a Wikipedia entry says more about RethinkDB or about Wikipedia?

coffeemug 11y ago
Slava @ Rethink here. There used to be a few Wikipedia articles that kept getting deleted. I don't really understand the Wikipedia guidelines on this, but I don't worry about it too much. As RethinkDB grows the article will get added back, and it will get harder to make an argument that it should be deleted.
- tptacek 11y ago
  There should be zero trouble getting a RethinkDB article written now, because it takes all of 5 seconds with NEWS.GOOGLE.COM to find reliable sources to back a notability claim. I can't find evidence of a deleted Rethink article in Wikipedia, but didn't look hard.
  - pests 11y ago
    Its listed right on the article's page: https://en.wikipedia.org/wiki/RethinkDB https://en.wikipedia.org/wiki/RethinkDB Reason: https://en.wikipedia.org/wiki/Wikipedia:Criteria_for_speedy_deletion#G11 https://en.wikipedia.org/wiki/Wikipedia:Criteria_for_speedy_... (G11. Unambiguous advertising or promotion)

mosselman 11y ago

So I have a question for you all. But first a little background. I have been playing with PouchDB, an in-the-browser implementation of CouchDB that is mostly compatible with CouchDB. What I really like is its sync function, this sets up no-brainer practically real-time sync between my local (offline) PouchDB database. It is very cool! I am very impressed by RethinkDB's cluster management, etc, so I would like to explore it as an option, but is there an easy way to sync my offline (browser based) localstorage-like database to rethink and back again? PouchDB makes this dead easy.

coffeemug 11y ago
There isn't a good way to do that yet. We've been playing with some ideas, but offline sync like this is a surprisingly challenging problem -- it's easy to make something that works, but dramatically harder to make something that works at scale.
- mosselman 11y ago
  I can imagine that. I have seen some sync implementations for Backbone.js for instance a while back, but nothing that really worked well. Again, I am very surprised of how well PouchDB performs in this. Additionally, you get all (most) the features of CouchDB in the browser, you can even use the design documents that you create on the server.
realusername 11y ago
Excuse my curiosity unrelated to the current topic but I'm about to deploy a small scale hybrid (desktop & web) application using PouchDB. I have strongly unreliable networks (3G networks with entire days where the app is offline). The nodes are owning their own chunk of data so there is no risk of conflicts whatsoever, the main goal is to sync as soon as they can. PouchDB/CouchDB seemed clearly the best fit for this kind of unusual application. Did you encounter any problem with it ? Or if you can share your opinion on this technology after using it.
- mosselman 11y ago

zak_mc_kracken 11y ago

It's a bit worrisome that there are no official Java drivers and the only unofficial Java project has been abandoned by its only author because he no longer uses Rethink.

danielmewes 11y ago
Daniel @ RethinkDB here. We're going to ship an official Java driver very soon. We decided to focus on a small number of core drivers first while the protocol and query language were still undergoing rapid extension. Now that the protocol is stable, we're going to expand our official driver support step by step. The Java one will be first. You can follow the progress on https://github.com/rethinkdb/rethinkdb/issues/3930 https://github.com/rethinkdb/rethinkdb/issues/3930
- e2e4 11y ago
  Thats great news. I love RethinkDB; but not having a Java driver made it a no-go for a couple of projects.
habitue 11y ago
Hey there, I am literally working on the official Java driver right now, fear not!

peterbe 11y ago

I added a RethinkDB benchmark when used with the Python Tornado web server: http://www.peterbe.com/plog/fastestdb http://www.peterbe.com/plog/fastestdb Scroll down for the update.

danielmewes 11y ago
Note that (according to the linked commits), RethinkDB is using what we call "hard" durability in this comparison. This is our default to ensure maximum data safety. Hard durability means that every individual write will wait for the data to be written to disk before the next one is run (in this benchmark, since it only does one at a time). I don't think any of the other databases in this test is using a similarly strict requirement, are they? You'd have to run with the currently commented line "rethinkdb.db('talks').table_create('talks', durability='soft').run(conn)" to get more comparable results. (Edit for clarification: `durability='soft'` is comparable to the `safe` flag in many of the MongoDB drivers. It means that the server will acknowledge each write when it has been applied, but not wait for disk writes to complete.)

dcre 11y ago

I imagine the One Direction reference will be lost on most readers.

jfroma 11y ago

There is some FUD in this blogpost as well as in the comments in this thread. I think the current status quo for databases is canned software. And this isn't necessarily bad because neither of the three databases mentioned hide their specs or default settings, the three have very good docs and community willing to help, in addition to companies giving commercial support. Whats your excuse to misuse these products? RethinkDB writes your data to disk before acknowledging the write but on the other hand can't elect a new primary in case of failure, two completely different features/limitations that might work for someone and not for other ones. Is that hard to understand? Did mongo documentation lie you at some point? Accept that you are "buying" a general purpose product, the designers thought that their users will need those features, deal with it. Otherwise build your own database, I know this might sound very hard but I guess in the future we will see smaller building blocks that let you build something that handle your needs like this: https://github.com/rvagg/node-levelup/wiki/Modules#plugins https://github.com/rvagg/node-levelup/wiki/Modules#plugins

NhanH 11y ago
I was just thinking about this issue in the last few days. I'm working on a side project, due to the nature of the data model, converting back and forth to fit into a relational database is kind of annoying, so I was looking around for other databases. Right now, the situation with database is that we have to convert our internal data structure into a representation that fit the data model of the database we're using (ie rows for relational, document/key for the NoSQL group). I can see the reason the data model has to be that way for scaling, distributed etc... But if I'm happy to scale my database up, and would prefer to have the database storing the data as close to the memory data structure as possible (similar to object databases -- albeit with a boarder definition of "object"), is there any database that could do that? Otherwise, is there any suggestion on how I could get started to build one?
- mrkurt 11y ago
  Redis is good at storing various data structures. Sets, sorted sets, hashes, etc. As long as "scaling up" means adding memory it's good.
11y ago

hoodoof 11y ago

There a common form of such posts by developers: "I've found database X and I LOVE it, we're migrating all our systems." Then a year later "Things didn't work out, here's how we migrated from database X to database Y." Someone should make an index of such blog posts.

sanguy 11y ago

The concern I have is that RethinkDB announced their death and vanished for 9+ months leaving everyone in a lurch, before resurfacing. The way they abandoned their users was deplorable so why trust them again? If you are into S&M just buy Oracle licenses...

antonmaju 11y ago
May I know when that happened? I'm curious about using RethinkDB after reading Rob's post
coffeemug 11y ago
Slava @ Rethink here. I'm not entirely sure what you're referring to, but the closest I can think of is the move from the memcached interface in 1.1 to the ReQL interface in 1.2. If that's what you mean, I don't think it's fair to say we abandoned our users at all. The memcached interface had very few people using it (literally single digits). We tried really hard to make it work, but there just wasn't any demand, so we decided to add a full query language, clustering, and rebuild with the realtime architecture. We supported the binary for a while, and helped most of the users migrate from the memcached interface to ReQL (which was fairly easy). We also helped people migrate to other memcached alternatives if they chose to not to use ReQL. In almost all cases people could quite literally pick another compatible product without changing any of their code. We took our time, helped people migrate (either to the newer version of RethinkDB, or to other products), and integrated the original architecture and as much of the code as we could into the new and improved RethinkDB. All of this was completely free of charge. So respectfully, I really don't think you're being fair to us. I'm sorry if this inconvenienced your company, but given the dire circumstances at the time we really did the best we could (and arguably, much, much more than most companies do in those circumstances).

pbz 11y ago

What do you guys think of Aerospike (http://www.aerospike.com/ http://www.aerospike.com/)? I heard of a few cases that used it successfully. Any issues with it?

chupy 11y ago
Aerospike - distributed key value store has a little bit different of a use case than RethinkDB - distributed JSON document databases. Basically the only valid use that I have seen for Aerospike, and that is the one that they are advertising is a distributed key value profile store for Ad-tech or marketing companies (http://www.aerospike.com/overview/ http://www.aerospike.com/overview/). Keep in mind that to actually use all of Aerospike's features , especially the one they are really proud about - the cross datacenter replication - you need the commercial license. So..I suggest you figure out what your requirements are and then use the best tool for the job.
- bbulkow 11y ago
  CTO at Aerospike here. Aerospike is battle-tested in large deployments --- ad-tech, marketing-tech, a few new ones in telecom and fin-serv. Pushing huge load with very, very little downtime. That's what we're the most proud of --- and I'm proud that we're able to offer this killer codebase as open source, after being closed source for the first few years of the company. Most applications have a huge core of key-value --- twitter, for example --- and need a fast and scalable key-value component. You can start with a single server (on your laptop with Vagrant) and scale up later. We're adding more types, more cool operations, more indexes this year. The fact that Aerospike has a basic query system, type safety, flash optimization (Amazon has switched over to being very SSD/Flash centric) support for every language under the sun (three contributed Scala layers --- and we see a lot of Go use as well as the usual Java / Node / Python / PHP / HHVM), Hadoop integration....

beefsack 11y ago

Have used RethinkDB with a few projects now, mainly in Go with dancannon's Go driver[1], and have found it very functional and robust and has become my go-to schemaless database. My largest project has been running a couple of years now and has accumulated a significant amount of data, and RethinkDB hasn't had any trouble at all scaling with my data growth. I'm running it on servers below the recommended requirements too (512MB DO instances) and have been really impressed with how it handles constrained resources. [1] https://github.com/dancannon/gorethink https://github.com/dancannon/gorethink

timmaxw 11y ago

Note that the 9ms performance claim in this article is mistaken. The author has issued a correction: http://rob.conery.io/2015/04/17/rethinkdb-2-0-is-amazing/ http://rob.conery.io/2015/04/17/rethinkdb-2-0-is-amazing/